[LiVE] Quantile Regression Workflows (WL Live-Stream Series)

Ideas
, , , , ,
1

The live-coding sessions

  1. In the first
    live-streaming / live-coding session
    I demonstrated how to make
    Quantile Regression
    workflows using the software monad QRMon
    and some of the underlying software design principles. (Namely
    “monadic programming” .)

  2. In the follow up live-coding session I discussed topics like outliers removal (data cleaning), anomaly detection, and structural breaks.

  3. In the third live-coding session:

    • First, we demonstrate and explain how to do QR-based time series simulations and their applications in Operations Research.
    • Next, we discuss QR in 2D and 3D and a related application.

  4. In the fourth live-coding session we discussed the following the topics.

    • Review of previous sessions.
    • Proclaiming the upcoming ResourceFunction["QuantileRegression"].
    • Predict tomorrow from today’s data.
    • Using NLP techniques on time series.
    • Generation of QR workflows with natural language commands.

  5. In the final, fifth live-coding session different questions are going to be answered.

    • Can we apply Quantile Regression (QR) without the package QRMon?
      • Yes, see the Wolfram Functions Repository item “QuantileRegression”.
    • Are you computing QR using a moving window?
      • Related, how QR compares to Local regression? Or LOESS?
    • What do you do when the QR fitted curves (regression quantiles) intersect?
    • How does QR fitted curve looks like over 3 points?
    • What approximation to pick for reconstructing the conditional CDF’s?
    • Why use QR? Is it just for better visualization of the signal?
    • What is the point of using those anomaly detection methods – a human can easily do it?
    • Can we use Neural Networks instead of QR?
    • Are there implementations in other popular DS languages?

Notebooks

  • I did not save the notebook I made during the first live coding session, but I attached to this post a modified version of the notebook I used for a Meetup workshop 6-7 months ago. (See for more details
    the GitHub MathematicaVsR project “Quantile Regression Workflows”.)

  • The notebook of the 2nd session is also attached. (I added a “References” section to it.)

Resource function

Instead of the software monad QRMon used in the live-coding sessions the Wolfram Resource Function
QuantileRegression
can be used. The resource page has many example applications that are also discussed in the live-coding sessions.

QRMon-Live-coding-session-5th-session-LIVE.nb (1.34 MB)
QRMon-Live-coding-session-examples-2nd-session-LIVE.nb (1.01 MB)
QRMon-Live-coding-session-examples-3rd-session-LIVE.nb (9.55 MB)
QRMon-Live-coding-session-examples-4th-session-LIVE.nb (1.41 MB)

6 Likes
2

Great job Anton! Looking forward to your future live-streams (next one will be today Sep 13 at 1:30pm PT, Wolfram TwitchTV channel)

1 Like
3

Just attached/uploaded the “live” notebook of the second session.

1 Like
4

Join for the 3rd session now: Twitch

1 Like
5

Just added/attached the notebook of the 3rd live-coding session.

6

Join us for the 4th session on Tuesday Oct. 22nd, 2pm PT (Twitch)
4th session: Quantile Regression workflow: additional topics.

  • Predictions for today from yesterday’s data.
  • Conversational agent generated workflows.
  • Application of NLP techniques to time series processed with QRMon.
1 Like
7

Just uploaded the notebook from the 4th live-coding session.

8

A few days ago the
Wolfram Function Repository entry [QuantileRegression] (https://resources.wolframcloud.com/FunctionRepository/resources/QuantileRegression)
was approved.
The resource description page has many of the topics discussed in the live-coding sessions on Quantile regression.

Here is the code result of the neat examples section:

SeedRandom[20];
distData = Table[{x, Exp[-x^2] + RandomVariate[NormalDistribution[0, .15 \[Sqrt](Abs[1.5 - x]/1.5)]]}, {x, -3, 3, .01}];
Grid[Table[
  Block[{tempData = RandomSample[distData, 100], data, probs}, 
   data = Join[tempData, {#[[1]], -#[[2]]} & /@ tempData];
   data = SortBy[Transpose[Rescale /@ Transpose[data]], #[[1]] &];
   probs = {0.02, 0.48, 0.52, 0.98};
   qFuncs = 
    ResourceFunction["QuantileRegression"][data, 5, probs, Method -> {LinearProgramming, Method -> "InteriorPoint",Tolerance -> 10^(-2)}]; 
   ListPlot[Prepend[(Transpose[{data[[All, 1]], #1 /@ data[[All, 1]]}] &) /@ qFuncs, data], 
   Joined -> Prepend[Table[True, Length[probs]], False], 
    Filling -> Prepend[Table[(i + 1) -> {i + 2}, {i, Length[probs] - 1}], 1 -> None], Frame -> True, FrameTicks -> False, Axes -> False, 
    ImageSize -> Small, AspectRatio -> 1]], 3, 3]]

QR-NeatExample.png
QR-NeatExample.png580×566 67 KB

3 Likes
9

Great work, Anton. Having the ResourceFunction available should help a lot of people.

3 Likes
10

Thank you, Seth! WFR is a great way to introduce new, user derived functions in the way, say, CRAN does for R.

2 Likes
11

Just uploaded the notebook from the final, 5th live-coding session.

12

YouTube playlist of lectures, so far

1 Like