We use technical cookies and Zorky's de-identified visit log. Yandex Metrica (data is stored on servers in Russia) runs only with your consent. We never sell data. More about cookies and statistics.
Zorky is in test mode — everything is free. Jobs and company pages are open to everyone; links to postings and your personal feed open after you sign up. Sign up →

Big Data: market

Data as of November 19, 2026: 508 open Big Data jobs, 31% remote among jobs that state a work format; median advertised salary $6,825/mo (from 61 postings that state a salary). Source: Zorky, updated daily.

Big Data Engineer — processing very large data volumes (terabytes, petabytes) via distributed engines. A Data Engineering sub-segment with focus on Spark/Hadoop/Flink/distributed computing. A premium niche thanks to distributed-systems complexity. Role family: Spark Engineer (PySpark or Spark Scala — Big Data's main application), Hadoop Engineer (legacy enterprise — HDFS + Hive + MapReduce, migrations to Spark), Flink Engineer (true streaming, low-latency event processing), Big Data Platform Engineer (Hadoop cluster admin, Spark on k8s + Iceberg/Delta), Data Lakehouse Architect (modern architecture — Iceberg/Delta + Spark + dbt). Stack: Apache Spark (must — PySpark or Scala, execution model, shuffles, broadcasts), Hadoop (HDFS + Hive + YARN — legacy in large enterprise), Apache Flink (streaming, premium niche), Kafka (must), Delta Lake/Apache Iceberg (lakehouse table formats — modern standard), Apache Hudi (lakehouse alternative), HBase/Cassandra (NoSQL distributed for real-time), Scala or Python (Scala for performance-critical Spark, Python for most), Airflow (orchestration), Cloudera/Hortonworks (distributions, migrations to the open-source stack), Kubernetes (Spark on k8s — modern deployment). Top stack: python, spark, cloud, java, hadoop. Among jobs with a stated work format, 31% are remote.

Data as of:

Key numbers for the Big Data market: open jobs, median salary, observed talent pool and remote share.

508
open jobs — by job title, reposts merged
$6,825
median $/mo
31%
remote

The Big Data market currently has 508 open roles, 442 of them freshly observed. Median salary $6,825/mo.

Among jobs with a stated work format, 31% are remote. Cluster admin work — cloud-based. International cloud-data SaaS (Databricks/Snowflake/Confluent) — full-remote.

⚠ salary known for 61 of 508 jobs; remote share from 354 with a stated format; 442 counted as fresh observations; trend and hiring difficulty are not shown

Timeline: postings and sources

How many Big Data postings appeared each week since Jun 1, and from how many sources. The lower chart separates market growth from coverage growth: a jump in the week new sources were connected means we started seeing more, not that the market grew.

New postings per week

Chart data
New postings per week. Block data table. A missing value is not zero; units and sample are stated in the metric description.
PeriodNew postings per week
2026-06-0148
2026-06-0826
2026-06-1525
2026-06-2286
2026-06-2983
2026-07-0669
2026-07-13128
2026-07-20135
2026-07-2767
2026-08-0356
2026-08-1028
2026-08-1729
2026-08-2425
2026-08-3156
2026-09-0737
2026-09-1439
2026-09-2181
2026-09-2891
2026-10-0598
2026-10-12342
2026-10-19133
2026-10-26117
2026-11-02106
2026-11-09134

Active sources per week

Sources over the period: 131. ▲ — a week with a large batch of new sources.

Chart data
Active sources per week. Block data table. A missing value is not zero; units and sample are stated in the metric description.
PeriodActive sources
2026-06-0113
2026-06-0812
2026-06-1511
2026-06-2224
2026-06-2927
2026-07-0635
2026-07-1328
2026-07-2033
2026-07-2724
2026-08-0330
2026-08-1018
2026-08-1719
2026-08-2414
2026-08-3122
2026-09-0719
2026-09-1417
2026-09-2127
2026-09-2834
2026-10-0527
2026-10-1236
2026-10-1935
2026-10-2632
2026-11-0234
2026-11-0948
New sources connected: Jun 22 +14 · Jun 29 +13 · Jul 6 +18 · Jul 13 +8 · all weeks (23) in the table
WeekNew sourcesNew postings
Jun 8+626
Jun 15+325
Jun 22+1486
Jun 29+1383
Jul 6+1869
Jul 13+8128
Jul 20+5135
Jul 27+667
Aug 3+656
Aug 10+128
Aug 17+229
Aug 24+125
Aug 31+456
Sep 7+137
Sep 14+139
Sep 21+381
Sep 28+391
Oct 5+198
Oct 12+6342
Oct 19+5133
Oct 26+2117
Nov 2+2106
Nov 9+7134

The current week is not over yet and is not shown: an incomplete week would look like a drop.

Week = posting first-seen date, all statuses; postings are selected by the same rule as this page's counter. Sources are distinct collection channels; their names are not published.

Chart data
Timeline: postings and sources — Big Data. Block data table. A missing value is not zero; units and sample are stated in the metric description.
WeekNew postings per weekActive sources
2026-06-014813
2026-06-082612
2026-06-152511
2026-06-228624
2026-06-298327
2026-07-066935
2026-07-1312828
2026-07-2013533
2026-07-276724
2026-08-035630
2026-08-102818
2026-08-172919
2026-08-242514
2026-08-315622
2026-09-073719
2026-09-143917
2026-09-218127
2026-09-289134
2026-10-059827
2026-10-1234236
2026-10-1913335
2026-10-2611732
2026-11-0210634
2026-11-0913448

Salaries over time

Median Big Data salary per week, USD per month.

Median salary, USD per month

Chart data
Median salary, USD per month. Block data table. A missing value is not zero; units and sample are stated in the metric description.
PeriodMedianPostings with a salary per week (n)
2026-07-205,75012
2026-07-27—4
2026-08-03—0
2026-08-10—0
2026-08-17—0
2026-08-24—0
2026-08-31—0
2026-09-07—0
2026-09-14—1
2026-09-21—4
2026-09-28—3
2026-10-055,07910
2026-10-126,79320
2026-10-196,25036
2026-10-266,72621
2026-11-027,69922
2026-11-094,65822

The series starts on Jul 20: there is no data before that.

Postings with a salary per week (n)

Only postings with a USD salary and a known period (month or year), $500–$50,000 per month. Weeks with fewer than 10 such postings are left blank — that means no data, not zero.

Chart data
Salaries over time — Big Data. Block data table. A missing value is not zero; units and sample are stated in the metric description.
WeekMedian
2026-06-01—
2026-06-08—
2026-06-15—
2026-06-22—
2026-06-29—
2026-07-06—
2026-07-13—
2026-07-205,750
2026-07-27—
2026-08-03—
2026-08-10—
2026-08-17—
2026-08-24—
2026-08-31—
2026-09-07—
2026-09-14—
2026-09-21—
2026-09-28—
2026-10-055,079
2026-10-126,793
2026-10-196,250
2026-10-266,726
2026-11-027,699
2026-11-094,658

Remote work over time

Share of Big Data postings offering fully remote work, per week.

Remote postings, %

Chart data
Remote postings, %. Block data table. A missing value is not zero; units and sample are stated in the metric description.
PeriodRemote
2026-06-0179.1
2026-06-0831.6
2026-06-1511.8
2026-06-2231.5
2026-06-2923.9
2026-07-0632.4
2026-07-1344
2026-07-2075.6
2026-07-2768
2026-08-0320
2026-08-1025
2026-08-1728
2026-08-2420
2026-08-3120
2026-09-0717.1
2026-09-1423.7
2026-09-2137.5
2026-09-2836.9
2026-10-0543.6
2026-10-1236.8
2026-10-1931.9
2026-10-2625.9
2026-11-0225.3
2026-11-0924.6

The share is among postings that state a format (remote, hybrid, office). Weeks with fewer than 10 such postings are left blank.

Chart data
Remote work over time — Big Data. Block data table. A missing value is not zero; units and sample are stated in the metric description.
WeekRemote
2026-06-0179.1
2026-06-0831.6
2026-06-1511.8
2026-06-2231.5
2026-06-2923.9
2026-07-0632.4
2026-07-1344
2026-07-2075.6
2026-07-2768
2026-08-0320
2026-08-1025
2026-08-1728
2026-08-2420
2026-08-3120
2026-09-0717.1
2026-09-1423.7
2026-09-2137.5
2026-09-2836.9
2026-10-0543.6
2026-10-1236.8
2026-10-1931.9
2026-10-2625.9
2026-11-0225.3
2026-11-0924.6

Technologies over time

How the share of Big Data postings mentioning each of the top technologies changed.

Postings mentioning the technology, %

Chart data
Postings mentioning the technology, %. Block data table. A missing value is not zero; units and sample are stated in the metric description.
PeriodPythoncloudSQL
2026-08-1034.84.313
2026-08-1744.47.422.2
2026-08-24———
2026-08-314915.735.3
2026-09-0748.313.841.4
2026-09-1428.625.720
2026-09-2136.430.337.9
2026-09-2843.24237
2026-10-0557.847.851.1
2026-10-123332.627.7
2026-10-1961.441.247.4
2026-10-2659.444.850
2026-11-025028.434.1
2026-11-0954.524.237.4
Up to 5 on the chart:

Share among postings with a tagged stack; the series starts on Aug 10: earlier stack tags were assigned differently. Weeks with fewer than 20 tagged postings are left blank.

Chart data
Technologies over time — Big Data. Block data table. A missing value is not zero; units and sample are stated in the metric description.
WeekPythonJavacloudhadoopSQL
2026-08-1034.817.44.339.113
2026-08-1744.411.17.43.722.2
2026-08-24—————
2026-08-314913.715.715.735.3
2026-09-0748.320.713.817.241.4
2026-09-1428.625.725.717.120
2026-09-2136.424.230.327.337.9
2026-09-2843.222.2422137
2026-10-0557.835.647.823.351.1
2026-10-12333932.654.327.7
2026-10-1961.434.241.23647.4
2026-10-2659.431.244.815.650
2026-11-02503328.413.634.1
2026-11-0954.536.424.226.337.4

Countries over time

How the share of Big Data postings in each of the top countries changed, per week.

Postings in the country, %

Chart data
Postings in the country, %. Block data table. A missing value is not zero; units and sample are stated in the metric description.
PeriodPolandUSAIndia
2026-06-0162.2200
2026-06-08162024
2026-06-1514.333.319
2026-06-2257.58.815
2026-06-2912.52015
2026-07-0614.332.712.2
2026-07-1322.328.720.2
2026-07-2053.27.310.5
2026-07-2751.612.96.5
2026-08-0313.719.621.6
2026-08-1017.417.430.4
2026-08-173.826.919.2
2026-08-2402015
2026-08-311.913.218.9
2026-09-0711.111.116.7
2026-09-146.221.918.8
2026-09-2122.132.414.7
2026-09-2811.926.211.9
2026-10-0546.49.53.6
2026-10-1214.533.69.3
2026-10-193220.36.2
2026-10-2626.315.815.8
2026-11-0214.721.110.5
2026-11-0919.810.75
Up to 5 on the chart:

Share among postings with a known country; the chart shows the five countries with the most postings over six months. Weeks with fewer than 20 postings with a known country are left blank. A rising share can mean a newly connected source in that country rather than market growth.

Chart data
Countries over time — Big Data. Block data table. A missing value is not zero; units and sample are stated in the metric description.
WeekPolandUSAIndiaSingaporeSouth Africa
2026-06-0162.220000
2026-06-08162024012
2026-06-1514.333.31909.5
2026-06-2257.58.81500
2026-06-2912.5201538.81.2
2026-07-0614.332.712.210.20
2026-07-1322.328.720.22.10
2026-07-2053.27.310.512.10.8
2026-07-2751.612.96.516.11.6
2026-08-0313.719.621.63.95.9
2026-08-1017.417.430.400
2026-08-173.826.919.23.80
2026-08-2402015100
2026-08-311.913.218.91.915.1
2026-09-0711.111.116.7013.9
2026-09-146.221.918.83.112.5
2026-09-2122.132.414.75.90
2026-09-2811.926.211.916.74.8
2026-10-0546.49.53.64.88.3
2026-10-1214.533.69.31.912.6
2026-10-193220.36.277
2026-10-2626.315.815.86.14.4
2026-11-0214.721.110.512.67.4
2026-11-0919.810.755.818.2

Clean market: hiring index

The same 7,505 employer job boards; the posting count comes from the board itself. Connecting new sources does not move this index — it shows the market, not our coverage. This is the whole panel market, not just Big Data.

Hiring index, Oct 16 = 100

Chart data
Hiring index, Oct 16 = 100. Block data table. A missing value is not zero; units and sample are stated in the metric description.
PeriodHiring index
2026-10-16100
2026-10-17100.03
2026-10-1899.76
2026-10-1999.68
2026-10-2099.27
2026-10-2198.6
2026-10-2299.07
2026-10-2397.4
2026-10-2497.41
2026-10-2697.33
2026-10-2796.76
2026-10-2896.21
2026-10-2995.17
2026-10-3095.13
2026-10-3193.65
2026-11-0193.33
2026-11-0292.67
2026-11-0392.65
2026-11-0492.41
2026-11-0592.13
2026-11-0690.44
2026-11-0790.48
2026-11-0890.45
2026-11-1090.29
2026-11-1190.22
2026-11-1390.12
2026-11-1489.7
2026-11-1589.68
2026-11-1689.62
2026-11-1888.91
2026-11-1988.02

Chain index: each day compares only boards polled on both that day and the previous one; outliers and platform-wide polling outages are excluded. Days with no comparable pairs are left blank.

Chart data
Clean market: hiring index — Big Data. Block data table. A missing value is not zero; units and sample are stated in the metric description.
DayHiring index
2026-10-16100
2026-10-17100.03
2026-10-1899.76
2026-10-1999.68
2026-10-2099.27
2026-10-2198.6
2026-10-2299.07
2026-10-2397.4
2026-10-2497.41
2026-10-2697.33
2026-10-2796.76
2026-10-2896.21
2026-10-2995.17
2026-10-3095.13
2026-10-3193.65
2026-11-0193.33
2026-11-0292.67
2026-11-0392.65
2026-11-0492.41
2026-11-0592.13
2026-11-0690.44
2026-11-0790.48
2026-11-0890.45
2026-11-1090.29
2026-11-1190.22
2026-11-1390.12
2026-11-1489.7
2026-11-1589.68
2026-11-1689.62
2026-11-1888.91
2026-11-1988.02

How many open Big Data jobs there are and how many matching specialists Zorky observes.

Demand and observed supply

Chart data
Block data table. A missing value is not zero; units and sample are stated in the metric description.
MetricValue
Open demand508
Open demand508
Observed supply— not published

Big Data forms a premium distributed-systems segment. Drivers: legacy Hadoop → Spark on k8s + Iceberg migrations, demand for streaming (Flink + Kafka), growth of Databricks partners. Trend 2026: lakehouse architecture (Iceberg/Delta) dominates, Hadoop EOL in large enterprise.

⚠ candidates matching a vacancy are not counted yet: demand and the observed pool are shown

How salaries in Big Data jobs are distributed, in US dollars per month.

Salary distribution

P25 $5,375 · P50 $6,825 · P75 $9,250
Chart data
Block data table. A missing value is not zero; units and sample are stated in the metric description.
MetricValue
<$1k0
$1–2k4
$2–3k2
$3–4k2
$4–5k5
$5–6k7
$6–7k12
$7–8k8
$8–9k5
$9–10k4
$10–12k3
$12–15k3
$15–20k5
$20k+1
Salary in postings: base 508
  • usable for the calculation: 61 (12%)
  • stated but not converted to USD/month: 63 (12.4%)
  • not stated: 384 (75.6%)

One in ten postings offers under $3,289, one in ten over $14,641. Half of the postings offer between $5,375 and $9,250. Sample: 61.

Chart data
Salary distribution — Big Data. Block data table. A missing value is not zero; units and sample are stated in the metric description.
MetricLower boundUpper boundCount
<$1k01,0000
$1–2k1,0002,0004
$2–3k2,0003,0002
$3–4k3,0004,0002
$4–5k4,0005,0005
$5–6k5,0006,0007
$6–7k6,0007,00012
$7–8k7,0008,0008
$8–9k8,0009,0005
$9–10k9,00010,0004
$10–12k10,00012,0003
$12–15k12,00015,0003
$15–20k15,00020,0005
$20k+20,000—1

Which countries have the most open Big Data jobs.

Demand geography

countryjobsshare
Poland11327.2%
United States8620.7%
India4310.3%
Singapore286.7%
Canada174.1%
United Kingdom153.6%
Russia133.1%
France112.6%
Spain102.4%
Germany92.2%
Other countries7117.1%

The leader by Big Data job count is Poland (113 positions). Moscow dominates. Poland — Big Data-friendly EU market. Large international remote via Databricks/Snowflake.

⚠ job counts only: salary by country is not published

Chart data
Demand geography — Big Data. Block data table. A missing value is not zero; units and sample are stated in the metric description.
CountryJobs
PL113
US86
IN43
SG28
CA17
GB15
RU13
FR11
ES10
DE9

Which technologies are most often required in the same job as Big Data.

Used together with

python
21742.7%
spark
21542.3%
java
16332.1%
cloud
15730.9%
hadoop
15129.7%
sql
14228%
aws
11823.2%
pyspark
10520.7%
databricks
9218.1%
azure
7715.2%

Share — how many of the 508 jobs in the Big Data market mention this technology. Ordered by job count, not by “strength of association”. Umbrella terms like cloud are not a specific cloud (AWS, Azure).

Top Big Data stack 2026: Apache Spark (must — PySpark or Scala), Hadoop (HDFS + Hive — legacy enterprise), Apache Flink (premium streaming), Kafka (must), Delta Lake / Apache Iceberg (lakehouse — modern standard), Apache Hudi, HBase / Cassandra (NoSQL distributed), Scala or Python, Airflow, Cloudera / Hortonworks, Kubernetes (Spark on k8s). Senior — Apache Atlas / DataHub.

Common pairs: Spark + Kafka, Spark + Iceberg, Hadoop + Hive, Spark + Scala, Flink + Kafka. Learning roadmap: Data Engineer Middle experience → Spark deeply → Spark execution model → one lakehouse (Iceberg) → Hadoop basics (for legacy) → Spark on k8s.

Chart data
Used together with — Big Data. Block data table. A missing value is not zero; units and sample are stated in the metric description.
NameCount
python217
spark215
java163
cloud157
hadoop151
sql142
aws118
pyspark105
databricks92
azure77
scala74
kafka70

How many Big Data jobs there are at each level, from junior to principal.

Demand by grade

gradejobs
senior141
lead48
middle47
principal20
junior6

Juniors are almost non-existent — the market expects Data Engineer Middle experience. Career flow: Data Engineer Middle → Big Data Middle in 4-8 months → Senior → Platform Engineer / Lakehouse Architect / Head of Big Data.

⚠ demand side only: the grade of the observed pool is unknown for most of it

Chart data
Demand by grade — Big Data. Block data table. A missing value is not zero; units and sample are stated in the metric description.
LevelCount
senior141
lead48
middle47
principal20
junior6

Who hires the most for Big Data and how concentrated demand is among the largest companies.

Employers

Demand is spread across 293 employers. The largest accounts for 7.9%, the top ten for 20.5%; the remaining 79.5% is long tail.

largest 7.9%2nd–3rd 3.9%4th–10th 8.7%the rest 79.5%
sharevalue
top-17.9%
top-311.8%
top-1020.5%
long tail79.5%

⚠ names are not shown: staffing agencies and end employers are not yet told apart by the classifier

Which company sites and job boards Zorky collects Big Data jobs from.

Where Zorky sees this market

Observed across 64 sources; the largest accounts for 18.9% — this market does not rest on a single channel.

Chart data
Where Zorky sees this market — Big Data. Block data table. A missing value is not zero; units and sample are stated in the metric description.
ClassNameEnglish nameCountShare
atsATS и карьерные сайтыATS & career sites2570.5059
boardджоб-борды и агрегаторыjob boards & aggregators2400.4724
telegramTelegramTelegram110.0217

The newest Big Data jobs from company career sites.

Recent openings

All jobs →

Latest open Big Data jobs — the most recent 10 positions with adequate description quality. The full list is in our CRM or via the "see all" link below.

How this is measured
Vacancy
an open job that cleared the quality gate and lists at least two technologies
Observed candidate
a candidate whose stack contains this technology; an aggregate — not a single record leaves the perimeter
Matchable candidate
not counted yet
Window
jobs open at the moment the snapshot was built

About the data

  • Some breakdowns are hidden: their data coverage is not yet sufficient.
  • Statistics are shown only where the sample clears a quality gate.
  • A missing block does not mean a value of zero.

Breakdowns currently hidden: 8.

Direction: Data Engineering

Related specializations

Inside Big Data Engineer: Spark Engineer (PySpark or Spark Scala — main application), Hadoop Engineer (legacy enterprise — HDFS + Hive + MapReduce), Flink Engineer (true streaming, premium niche), Big Data Platform Engineer (Hadoop cluster admin, Spark on k8s + Iceberg/Delta), Data Lakehouse Architect (modern architecture). Comparison with other data specialisations — in the bar below.

Data EngineerDBAStreamingData Warehouse

Frequently asked questions

The most common questions about Big Data: pay, Spark vs Hadoop vs Flink, Lakehouse (Delta/Iceberg/Hudi), difference from Data Engineer / ML Engineer, remote, how to start (4-8 months after Data Engineer Middle), Senior skills. Answers recompute automatically.

What does a Big Data Junior, Middle, Senior, or Lead earn?

Junior Big Data openings are ALMOST NON-EXISTENT — the market expects Data Engineer Middle experience + hands-on Spark. The Junior → Middle jump — Spark execution model + one of Hadoop/Flink/Kafka deeply. Senior owns Big Data platform architecture + mentor. Lead Big Data — managing distributed systems with 5+ engineers, accountable for cluster sizing + cost optimisation. Career flow: Data Engineer Middle/Senior → Big Data Middle in 4-8 months → Senior → either Platform Engineer (infra-focus) or Lakehouse Architect (modern data stack).

What stack does Big Data most often need?

Top 5: python, spark, cloud, java, hadoop. Apache Spark — must (PySpark for most, Spark Scala for performance-critical). Deep understanding of the execution model (DAG, stages, tasks), shuffles, broadcasts, partitioning. Hadoop — HDFS + Hive + YARN — legacy enterprise (Sber/banks/telco), migrations to the open-source stack. Apache Flink — true streaming, premium niche. Kafka — must (partitions, consumer groups, exactly-once). Delta Lake/Apache Iceberg — lakehouse table formats (modern standard, replacing the classic Hadoop stack). Apache Hudi — alternative to Iceberg. HBase/Cassandra — NoSQL distributed for real-time. Scala or Python — Scala for production Spark, Python for most. Airflow for batch-job orchestration. Cloudera/Hortonworks distributions (legacy enterprise). Kubernetes — Spark on k8s + Spark Operator. Apache Atlas/DataHub — data lineage + catalog (Senior must). Knowledge of JVM tuning (GC G1/ZGC, heap profiling) — Spark Scala Senior must.

Spark vs Hadoop vs Flink — what to pick for Big Data?

Apache Spark — industrial standard + dominator. PySpark — the largest market, Spark Scala — premium niche. Downsides: micro-batch streaming (seconds latency), not true realtime. Hadoop (HDFS + MapReduce + Hive) — legacy enterprise. Used at large banks/telco where migration to open-source isn't finished. Must know if you work at Sber/banks. Trend: migrations from Hadoop to Spark on k8s + Iceberg/Delta Lake. Apache Flink — true streaming engine with millisecond latency, exactly-once semantics out of the box. Growing segment: FinTech/AdTech/IoT/real-time ML inference. Strategy: Spark first (market size), then Flink for diversification and premium pay. Learn Hadoop only if you're going into banking.

Lakehouse (Delta Lake / Iceberg / Hudi) — what to learn in 2026?

Lakehouse architecture — modern data architecture that combines Data Lake (cheap storage in S3/HDFS) with Data Warehouse (ACID + schema enforcement + indexing). Replaces the classic Hadoop stack. Three main table formats: Delta Lake (Databricks-original, most mature, dominator in Databricks shops), Apache Iceberg, Apache Hudi (third place, fastest updates via incremental ingestion). Strategy: Iceberg first (open-standard winner 2024–2026), Delta Lake if you work with Databricks, Hudi only for a specific use-case (CDC + updates-heavy).

What share of jobs are remote?

Among jobs with a stated work format, 31% are remote. Jobs without a stated format are excluded from the denominator; hybrid work is not counted as remote.

How is Big Data Engineer different from Data Engineer / ML Engineer?

Big Data Engineer (this page) — focus on distributed systems for processing terabytes+. Spark/Hadoop/Flink + cluster management. Premium for distributed-systems complexity. Data Engineer (general) — can work with smaller volumes (GB-TB) without Spark. Focus on pipelines + DWH modelling. ML Engineer — deploy models to production (FastAPI + ONNX/TorchServe), feature store, MLOps. Focus on inference. ML Engineer Senior ≈ Big Data Senior. Career switch Data Engineer → Big Data in 4-8 months (Spark + one lakehouse + Kafka). Big Data → ML Engineer in 8-12 months (PyTorch + MLOps stack). Many Senior Big Data engineers move into ML Engineering (data infrastructure → model deployment).

Where to start in Big Data in 2026?

Data Engineer Middle experience assumed (Python + SQL + Airflow + one DWH). Roadmap: 1) Apache Spark deeply — official documentation + Databricks Learning Academy (free). Master DataFrame API + Spark SQL + Window Functions + Structured Streaming. 2) Spark execution model — physical/logical plans, DAG, stages, tasks, shuffles, broadcasts, partitioning strategies. 3) PySpark — for most projects. Alternative: Spark Scala for performance-critical. 4) Kafka — partitions, consumer groups, exactly-once semantics. 5) One lakehouse format: Iceberg (recommended — open-standard winner) or Delta Lake (if Databricks). 6) Hadoop basics — HDFS + Hive (for maintenance of legacy banking projects). 7) Spark on Kubernetes (Spark Operator). 8) End-to-end pet project: Spark + Kafka + Iceberg + Airflow pipeline on a 10-50 GB dataset, deployed in the cloud. Courses: Karpov.Courses "Spark Developer", OTUS "Big Data Engineer", Yandex.Practicum (part of the Data Engineer track), Databricks Learning Academy (EN — best free resource), Coursera "Big Data" specialisation. Books: "Spark: The Definitive Guide" Chambers/Zaharia, "Designing Data-Intensive Applications" Kleppmann (must-read). Data Engineer Middle → Big Data Middle — 4-8 months.

How many open jobs are in this sample?

The published sample contains 508 open jobs after merging reposts. This is the same count shown in the main indicator. The geography section describes the sample coverage; this is not an estimate of the entire market.

What skills does a Senior Big Data Engineer need?

A Senior Big Data Engineer owns the full distributed-systems cycle. Spark mastery: execution model (logical plan → physical plan → tasks), shuffle optimisation (avoid shuffles where possible, broadcast joins for small tables, repartition vs coalesce, skew handling), Catalyst optimizer internals, Tungsten memory model, Spark SQL deep, Structured Streaming with exactly-once. Performance: JVM tuning for Spark Scala (G1/ZGC, heap sizing — executor.memory + driver.memory + memoryOverhead), pandas UDFs vs Pandas API on Spark, Photon engine (Databricks-only). Kafka: producer-consumer semantics deep, partitioning strategies, exactly-once via transactions, schema registry (Avro/Protobuf), Kafka Streams basics. Lakehouse: Iceberg/Delta — schema evolution, time-travel, partition evolution, optimize/vacuum, hidden partitioning. Cluster admin: Spark on k8s (Spark Operator), cost optimisation (spot instances, auto-scaling), Cloudera/Hortonworks for legacy. SQL: advanced SQL for Spark SQL + Hive SQL + one DWH (Snowflake/BigQuery). Data Governance: Apache Atlas or DataHub — lineage, catalog, access control. DevOps: Docker, Kubernetes, Terraform for IaC, CI/CD for Spark applications (Spark Submit + GitHub Actions). Soft: code review, mentoring, communication with Data Scientists / Analytics teams on requirements. English — must for Senior+ (Big Data documentation is predominantly EN, few Russian-speaking sources).

Leave a request

Describe the task and leave a contact — the request goes to our CRM and we reply at the contact you provide.

By submitting, you agree to the privacy policy.