1. Home
  2. Guides
  3. Concepts
  4. Tools
  5. References
    1. Apache Spark
    2. Built-in Functions
    3. Incorta Component SDK
    4. Incorta Connector SDK
    5. Connectors
    6. Copilot
    7. Data Studio
      1. Enabling the Data Studio
      2. Aggregation Recipe
      3. Change Type Recipe
      4. Data Quality Recipe
      5. Filter Recipe
      6. Formula Recipe
      7. Fuzzy Join Recipe
      8. Join Recipe
      9. LLM Recipe
      10. Python Recipe
      11. Rename Recipe
      12. Sample Recipe
      13. Save MV Recipee
      14. Select Recipe
      15. Sort Recipe
      16. Split Recipe
      17. SQL Recipe
      18. Union Recipe
      19. Unpivot Recipe
    8. Public API
    9. Security
    10. SQL Interface
    11. Visualizations
    12. Other
  6. Integrations
  7. Data applications
  8. Release Notes
  9. Incorta Premium

References → Python Recipe

The Python recipe is an excellent way to inject PySpark into your dataflow. This can be used for simple or complex operations that are not currently available as a recipe.

Configuration

ConfigurationDescription
Recipe NameA freeform name of how a user would like to name a recipe
InputSelect a previously constructed recipes. The code tool can ingest multiple inputs.
Change Column TypeEnter PySpark code to run within the recipe

Or, use Incorta Nexus to generate an intelligent code suggestion:
  ●   Enter the description or desired outcome.
  ●   Select the Incorta Nexus icon next to the description field.
  ●   View the generated PySpark code based on your input in the description field.

Important pySpark commands

Input Data

The inputs assigned in the Multiple Inputs setting will populate a new configuration section called Input Variable Name. This will showcase the true name of the underlying data frame. Users can copy these variable names as input data frames within their code.

Output Data

For each Code recipe, the code must contain output as a data frame for the recipe to save properly. Please use the function output_df() to save the data frame. Example: output_df(my_sample_df)