Rishitosh Kumar Singh

ML engineer. I build data pipelines and training workflows, and I research how LLM agents fail.

MS Computer Science, Arizona State University ’26CaliforniaAWS Certified

  • 5research papersEMNLP 2026 ×2, a NeurIPS 2026 workshop, 2 journals
  • 3+ yrsshipping ML in industryTiger Analytics and Techlearn, 2020 – 2023
  • 580agent tasks in my benchmarkτ-Emotion-Bench: 10 domains, 594 emotion conditions
  • 39%lower ETA errorSunTransit model vs a lookup-table baseline

Ten years at a glance

Experience

  1. Feb 2025 – PresentNow

    Research Assistant

    CogInt Lab & GLAD Lab, Arizona State University

    • 580verified agent tasks built
    • +19.7 ptsbest Avg@3 gain from Tracer fine-tuning
    • +18 ppVLM accuracy from few-shot retrieval
    LLM Agent BenchmarkVLM ResearchEMNLP 2026NeurIPS 2026 Workshop
    What I did
    • Built Tracer, the agentic data generator behind τ-Emotion-Bench: 580 verified tool-calling tasks across 10 domains with emotional simulated users. The best frontier agent reaches only 55.9% Avg@3; fine-tuning Qwen3 on 1,958 Tracer trajectories adds 13.8–19.7 points.
    • Co-first author on benchmarking VLMs (InternVL3.5, Qwen3-VL, GPT-4o) for reading player engagement from gameplay video across 9 games. Zero-shot scored below a majority baseline (~57% vs 67.2%); retrieval-augmented few-shot prompting closed the gap by +18pp (EMNLP 2026).
    • Built Dask and PySpark pipelines that clean, validate and normalize raw research datasets into Parquet for analysis and training.
    • Built 2 CrewAI multi-agent workflows that generate course assignments for AI/ML coursework.
  2. Jun 2022 – Dec 2023

    Machine Learning Engineer

    Tiger Analytics

    • 35%faster model retraining
    • 25%shorter batch runs
    • 60%less manual monitoring work
    MLOps on AWS SageMakerModel Drift DetectionBig Data ETL
    What I did
    • Built SageMaker pipelines for a financial client that chain data cleaning, inference, evaluation and drift detection: 35% faster retraining.
    • Built drift detection and retraining workflows with MLflow to keep production models reliable.
    • Tuned PySpark and StreamSets ETL over millions of daily records: 25% shorter batch runs.
    • Automated model monitoring and CI/CD: 60% less manual work.
  3. Jul 2020 – Jun 2022

    Senior Product Analyst (Applied ML)

    Techlearn (EduGrad)

    • 10% → 2%user churn
    • 62%lower inference cost
    NLP RecommendationsComputer VisionEdge Deployment
    What I did
    • NLP models on user behavior drove product recommendations that cut churn from 10% to 2%.
    • Shipped EduFace, a face recognition pipeline (GStreamer, OpenCV, Docker) on local hardware: 62% lower inference cost.
    • Used computer vision to split mentor recordings by topic, cutting manual review time.
    • Built a Flask platform for labeling and processing user and product data.

Projects

Research benchmark

τ-Emotion-Bench

Case study →PaperProject site

Do AI agents still get the job done when the customer is angry, scared or in a hurry? 580 tool-calling tasks in 10 domains, where the simulated user carries one of 594 emotion conditions and can walk away.

  1. Tool graphs
  2. Sampled traces
  3. User instruction
  4. Emotion mixed in
  5. Verified task

Explore the results

Even the best frontier agent, DeepSeek-V4.1-Flash, solves only 55.9% of tasks on average when the user is emotional.

  1. DeepSeek-V4.1-Flash 55.9DeepSeek-V4.1-FlashDeepSeek · non-thinkingAvg@3 55.9% · Pass^3 32.2%Loses 23.7 pts to inconsistency
  2. GLM-5.3 51.9GLM-5.3Z.ai · thinkingAvg@3 51.9% · Pass^3 31.7%Loses 20.2 pts to inconsistency
  3. Qwen3.8-27B 50.0Qwen3.8-27BQwen · thinkingAvg@3 50.0% · Pass^3 27.3%Loses 22.7 pts to inconsistency
  4. Kimi-K2.7-Code 49.3Kimi-K2.7-CodeMoonshot AI · thinkingAvg@3 49.3% · Pass^3 26.1%Loses 23.2 pts to inconsistency
  5. Gemma-4-31B-it 44.8Gemma-4-31B-itGoogle · thinkingAvg@3 44.8% · Pass^3 24.7%Loses 20.1 pts to inconsistency
  6. MiniMax-M2.7 38.6MiniMax-M2.7MiniMax · thinkingAvg@3 38.6% · Pass^3 16.8%Loses 21.8 pts to inconsistency
  7. Tracer-High-8B ours38.1Tracer-High-8Bτ-Emotion-Bench · thinkingAvg@3 38.1% · Pass^3 19.8%Loses 18.3 pts to inconsistency
  8. MiniMax-M3 37.0MiniMax-M3MiniMax · thinkingAvg@3 37.0% · Pass^3 17.6%Loses 19.4 pts to inconsistency
  9. Qwen3-235B-A22B (thinking) 36.0Qwen3-235B-A22B-Thinking-2507Qwen · thinkingAvg@3 36.0% · Pass^3 16.5%Loses 19.5 pts to inconsistency
  10. Muse-Glimmer-30B 35.1Muse-Glimmer-30BMeta · thinkingAvg@3 35.1% · Pass^3 20.5%Loses 14.6 pts to inconsistency
  11. Qwen3-235B-A22B (no thinking) 32.6Qwen3-235B-A22B-Instruct-2507Qwen · non-thinkingAvg@3 32.6% · Pass^3 13.8%Loses 18.8 pts to inconsistency
  12. Qwen3-32B (thinking) 28.5Qwen3-32BQwen · thinkingAvg@3 28.5% · Pass^3 8.6%Loses 19.9 pts to inconsistency
  13. Qwen3-32B (no thinking) 23.9Qwen3-32BQwen · non-thinkingAvg@3 23.9% · Pass^3 9.4%Loses 14.5 pts to inconsistency

Trained on our Tracer dataOther modelsAvg@3: mean success rate over three trials. Snapshot 2026-10-19. Full leaderboard ↗

PythonLLM evaluationFine-tuningQwen3

Data engineering + ML

SunTransit

suntransit.liveDocsCode

Real-time transit tracking for Phoenix and Boston from GTFS-RT feeds, with stop-level delay analytics. The ETA model cuts error (MAE) 39% vs a lookup-table baseline.

  1. GTFS-RT feeds
  2. Kafka
  3. Spark
  4. Redis + PostgreSQL
  5. Live API
  6. Map + ETAs

Live from the pipeline

Connecting to the live feed…

movingstopped or unknownevery dot is a real vehicle · hover one

–vehicles tracked right now
–routes in service
–daily average delay

SparkKafkaDockerSageMakerStreamlit

RAG chatbot

Richie

Open appDocsCode

A chatbot that answers questions about my work. It indexes 50+ GitHub repos into Pinecone, and LangGraph routes each query.

  1. Your question
  2. LangGraph router
  3. Pinecone search
  4. Grounded answer

Try it

Hi, I'm Richie. I've read Rishitosh's resume and 50+ of his GitHub repos. Ask me anything about his work.

LangGraphPineconeMongoDBStreamlit

Publications

Skills

LLM research
Tool-calling agents SFT (Qwen, LLaMA) Critique / reward modeling VLMs RAG Benchmark construction LLM-as-judge
ML / LLM
PyTorch Hugging Face Transformers scikit-learn LangChain LangGraph CrewAI FastMCP
Data
Spark Kafka Airflow Dask Snowflake Pandas SQL
Infra
AWS SageMaker S3 EC2 Lambda Docker Kubernetes MLflow FastAPI Flask
Languages
Python SQL C++ JavaScript Bash

Education

MS, Computer Science · Arizona State University2024 – 2026
BTech, Computer Science · Dr. A.P.J. Abdul Kalam Technical University2016 – 2020