PySpark crossJoin Explained: Building Every Product Variation for an E-Commerce Catalog

Sometimes a data task boils down to generating every possible combination of two lists. A clothing store might need to pair every fabric type with every fit style so the design team can plan the full product lineup. That’s exactly where PySpark’s crossJoin comes in, and it’s simpler than it looks.

The Scenario

Consider a data analyst at ThreadCraft, a clothing e commerce company. The product team has finalized two things for the upcoming season:

1. A list of fabrics they’ll be working with.
2. A list of fit styles each garment can come in.

The goal? Generate every single fabric and fit combination so the design team can sketch mockups and the inventory team can start planning warehouse space.

It starts with two small PySpark DataFrames:

fabrics_df = spark.createDataFrame(
    [("Linen",), ("Denim",), ("Cotton",), ("Corduroy",)],
    ["fabric"]
)

fits_df = spark.createDataFrame(
    [("Slim",), ("Regular",), ("Relaxed",)],
    ["fit"]
)

fabrics_df looks like this:

fabric
Linen
Denim
Cotton
Corduroy

And fits_df looks like this:

fit
Slim
Regular
Relaxed

The question is: how can every fabric be paired with every fit without writing a loop, without hardcoding anything, and in a way that scales to thousands of rows?

Enter cross

A cross join (also called a Cartesian product) takes every row from the first table and matches it with every row from the second table. If table A has 4 rows and table B has 3 rows, the result has 4 × 3 = 12 rows, one for every possible pair.

The PySpark syntax is refreshingly minimal:

catalog_df = fabrics_df.crossJoin(fits_df)
catalog_df.show()

Output:

+--------+-------+
|  fabric|    fit|
+--------+-------+
|   Linen|   Slim|
|   Linen|Regular|
|   Linen|Relaxed|
|   Denim|   Slim|
|   Denim|Regular|
|   Denim|Relaxed|
|  Cotton|   Slim|
|  Cotton|Regular|
|  Cotton|Relaxed|
|Corduroy|   Slim|
|Corduroy|Regular|
|Corduroy|Relaxed|
+--------+-------+

That’s it. One line of code, twelve product variations, zero loops.

What’s Actually Happening Under the Hood?

Think of it like a restaurant menu. If a restaurant offers 4 types of pasta and 3 types of sauce, the full menu of possible dishes is every pasta paired with every sauce. “Penne + Marinara” isn’t skipped just because “Penne + Alfredo” already exists. Every pairing gets listed.

crossJoin does the same thing. It doesn’t look for matching keys between the two tables (that’s what regular joins do). It simply says: give me every combination.

Making It More Useful

In practice, it often helps to enrich the result. For example, generating a product SKU for each combination:

from pyspark.sql.functions import concat, lit, upper, monotonically_increasing_id

catalog_df = (
    fabrics_df
    .crossJoin(fits_df)
    .withColumn(
        "sku",
        concat(
            upper(fabrics_df["fabric"]),
            lit("_"),
            upper(fits_df["fit"]),
            lit("_"),
            monotonically_increasing_id()
        )
    )
)

catalog_df.show(truncate=False)

+--------+-------+-------------------+
|fabric  |fit    |sku                |
+--------+-------+-------------------+
|Linen   |Slim   |LINEN_SLIM_0      |
|Linen   |Regular|LINEN_REGULAR_1   |
|Linen   |Relaxed|LINEN_RELAXED_2   |
|Denim   |Slim   |DENIM_SLIM_3      |
|...     |...    |...                |
+--------+-------+-------------------+

Now each combination has a unique identifier the warehouse team can actually use.

A Word of Caution: Size Matters

Cross joins multiply row counts. That’s fine when combining 4 fabrics with 3 fits, since the result is only 12 rows. But cross joining a table of 10,000 customers with a table of 5,000 products would suddenly produce 50 million rows. That can bring a cluster to its knees.

A good rule of thumb: if either DataFrame has more than a few thousand rows, it’s worth pausing to consider whether every combination is truly needed, or whether a filtered join would do the job.

When to Use crossJoin

Cross joins shine in a few specific situations:

1. Catalog generation pairs dimensions like fabric × fit, color × size, or topping × crust to build out a full product matrix.
2. Simulation and testing creates every combination of input parameters for a test harness or a hypothetical analysis.
3. Date scaffolding crosses a list of stores with a list of dates to build a skeleton table that will later be filled with sales data.
4. Pairwise comparison compares every item against every other item (think recommendation engines or distance calculations).

Wrapping Up

crossJoin is one of those PySpark methods that does exactly one thing and does it well. When every combination of two sets of data is needed, with no conditions, no key matching, just the full Cartesian product, it’s the right tool. Keeping an eye on row counts and using it intentionally will prevent performance headaches and eliminate the need for awkward nested loops or convoluted SQL.

For the ThreadCraft team, those 12 rows are now ready to hand off to the designers. Every fabric, every fit, every possibility accounted for in a single line of PySpark.