Referring to arguments of a function by order in which they appear

Questions
41

The basic problem is that we have two equivalent data structures that have different “shapes”. One of the shapes is a flat list and the other has some extra levels of structure. What I don’t know is if you have a bunch of different shapes rather than just two. Or maybe you have a bunch of scenarios that all have two shapes, but the two shapes are different across the pairs. So, situation #1 would be having, say 16 arguments that can show up in three different shapes: a flat list, a 4x4 matrix, and a 2x8 matrix. Situation #2 would be that for some particular function, we need to transform between a flat list and a 4x4 matrix, but for some other function we need to transform between a flat list and a 2x8 matrix. Anyway, what I would suggest is that you build helper functions to do the transformations and have your main functions leverage these helpers.

We know we can go from the more structured shape to a flat list with Flatten, so we only need to implement an Unflatten (or maybe several different Unflatten functions, in which case you should name them distinctly). It might also just help readability to be even more explicit about naming the argument patterns.

StructuredPattern = {{_, _}, {{_, _}}};
ListPattern = Flatten[StructuredPattern];

UnflattenSequence[a_, b_, c_, d_] := {{a, b}, {{c, d}}};
UnflattenList[list : ListPattern] := UnflattenSequence @@ list;

FunctionSequence[args___] :=
  <|"what I pass to the list processing function" -> List[args],
   "what I pass to the structure processing function" -> UnflattenSequence[args]|>;
FunctionList[args : ListPattern] :=
  <|"what I pass to the sequence processing function" -> Sequence @@ args,
   "what I pass to the structure processing function" -> UnflattenList[args]|>;
FunctionStructure[args : StructuredPattern] :=
  <|"what I pass to the sequence processing function" -> Sequence @@ Flatten[args],
   "what I pass to the list processing function" -> Flatten[args]|>;

FunctionSequence[w, x, y, z]
(*
  <|"what I pass to the list processing function" -> {w, x, y, z}, 
   "what I pass to the structure processing function" -> {{w, x}, {{y, z}}}|>
 *)

FunctionList[{w, x, y, z}]
(*
  <|"what I pass to the sequence processing function" -> Sequence[w, x, y, z], 
   "what I pass to the structure processing function" -> {{w, x}, {{y, z}}}|>     
 *)

FunctionStructure[{{w, x}, {{y, z}}}]
(*
  <|"what I pass to the sequence processing function" -> Sequence[w, x, y, z], 
   "what I pass to the list processing function" -> {w, x, y, z}|>
 *)
42

No need to pay.

43

Sorry, I am not familiar with associations, but I will learn if you can you try to implement your more general solution to this specific example, so that I can make fflat1 work, instead of having to type in all the dummy elements of all the lists as in fflat2 and then (double tedium) reconstructing them (I commented out all but the first list, because it is too tedious to type in all the dummy arguments, and will only get more tedious as I will have higher dimensional lists in the future)? Somehow, my attempts to make fflat1 receive a sequence with no structure did not work, and it still keeps the structure, as we can see with the failed first attempt to call fflat1 with a structureless sequence, and successful second attempt with a stuctured sequence which contains a list.

Open in Wolfram Cloud ↗
44

Sorry, I am not familiar with associations

Sorry for the distraction then. Associations aren’t inherent to the problem you’re trying to solve. I was just trying to demonstrate how to transform between representations.

I’ll try to refactor your code later. I only have a couple of minutes right now. So for now, I’ll just try to explain why it failed.

Here’s your definition for fflat1 (I’m removing the comments for clarity)

fflat1[Sequence @@ Flatten[{flag_, LRc_, cL_}]] := cL[[1]]

If you look at just the argument list you created, it looks like this.

Sequence @@ Flatten[{flag_, LRc_, cL_}]
(* Sequence[flag_, LRc_, cL_] *)

So, your function expects exactly three arguments. But when you tried to apply it:

fflat1[Sequence @@ Flatten[{1, 0.1, cout0L}]]

you actually provided 6 arguments:

Sequence @@ Flatten[{1, 0.1, cout0L}]
(* Sequence[1, 0.1, 4., 4., 4., 4.] *)

When you applied it the other way:

fflat1[1, 0.1, cout0L]

you did provide exactly three arguments:

Sequence[1, 0.1, cout0L]
(* Sequence[1, 0.1, {4., 4., 4., 4.}] *)

but the last argument was itself a list. So, with the argument constraint satisfied, the function was evaluated to be the first item in the third argument, which is 4..

45

Yes, I understand why it won’t evaluate that one line, but I was hoping you could figure out how to make it work in an elegant (non-tedious way).

Also, the whole reason for getting away from lists and giving ffoutsuper list-free arguments was initially that Derivative was not able to handle conditionals in lists, and those were only necessary in order to keep eqnlist unevaluated. But you figured out how to not evaluate it by having a dummy function in the definition of eqnlist, that only later gets substituted with the function I want (ffoutsuper).

However, though Derivative can handle lists (now without the hack conditionals), it takes a really long time to evaluate my actual ffoutsuper when we keep the lists. And moreover, there is another bug, which is that since my ffoutsuper needs to take the real part of certain eigenvalues of certain matrices, the derivative outputs such nonsense as Re’[number]. I have included a file that demonstrates both of these problems, for a simpler ffoutsuper than my actual one. If we could fix this, we probably wouldn’t need to do the whole list-> sequence rigamarole.

Open in Wolfram Cloud ↗
46

So, back to trying without lists in the args to ffoutsuper, since we Derivative has problems with lists in my real case, either being too slow, or trying to do nonsense like differentiaing the real part of a variable.
This should not be complicated, but I don’t know how to do it. I want to reconstruct the lists inside ffoutsuper. ffoutsuper gets a flat sequence, so the information of the list stuctures is lost to it, but not in my head. For each simulation I will have a constant list structure, though these will vary from simulation to simulation, and for now I don’t mind rewriting code for different simulations.

So I know where in the flat structure to start and where to end, for reconstructing all the lists, no matter what shape they have. I just need a way to refer to the nth argument of the flat sequence in the args of ffoutsuper. Can you show me how to do that? Do I need to define a pure function and use Slot? Here is a failed attempt to reconstruct one of the lists for a simple example
Wolfram Notebook

47

fflat2[args__] := (cL = args[[3 ;; 6]]; cL[[1]]^2 ); (how do I say cL=Table[third through sixth argument of fflat2?)

Since args is a Sequence, it doesn’t do what you expect when used as an argument to Part. The easiest way to fix this is to coerce it to a List:

fflat2[args__] := (cL = {args}[[3 ;; 6]]; cL[[1]]^2 )
48

Getting close, but still no cigar. I’m now able to give ffoutsuper a sequence, and resonstruct lists inside it (I only did it for 2 of the 2dimensional lists to keep it simple). But now I’m still getting superslow evaluation, and the Re’[number] bug. Any ideas?

Open in Wolfram Cloud ↗
49

And other issues that are seen in the actual code. Is there a way to tell Derivative to not even try to evaluate something symbolically?

50

If I understood what you’re trying to achieve mathematically, maybe I could give you suggestions, but I don’t, so I can’t. But as a Mathematica programming problem, you might be beating your head against a wall. I’m not sure that we can expect Mathematica to figure out the derivative that you want from the form that you’re providing to it.

Not that it really matters to the underlying problem, but you can avoid using temporary variables:

ffoutsuper[args__] := Max[Re[Eigenvalues[Dot @@ ArrayReshape[Drop[{args}, 10], {2, 4, 4}]]]]

I really don’t think I can help you if you can’t provide simpler examples. I mean, just forget your actual domain completely, and just show an example of dynamically generating a deferred derivative (or whatever we’re trying to do here). Show an example where you know the right answer and show me the right answer. The stuff you give me is too far along, and I don’t know where to start looking, and so I end up spending a lot of time reverse engineering your code. I’m not willing to do that any longer.

Some general advice: start small and take baby steps. Don’t get this far into your coding before you try to fix it. Start with the absolute simplest representation of your problem (or a minimal alternative to your problem) and just do one computation/transformation and make sure it works. Then build on that foundation with one more dead simple computation/transformation. You have so much crazy code here that I don’t know if you’ve made a straight up semantic error or if you’re just mis-using a Mathematica function or if we just need to change the order of evaluation. There may be a simple fix, or this whole thing might actually be impossible. I have no idea where we are on that spectrum.

51

I tried your ArrayReshape. It makes no difference either to the timing for Derivative (way too slow) or the bug with the Re’1. I am including the same file, but now instead of taking the highest eigenvalue of the product of two of the input matrices, it takes the highest eigenvalue of the product squared. Here it never stops the calculation, even after 20 minutes. There is something very wrong with Derivative, and I want you to admit it, instead of putting it on me to come up with something simpler (are you working for Wolfram?). It might be possible to simplify further the list of args, but why? It is much simpler than my actual application already. Derivative has a problem, maybe it can only take so many variables? Also, please take a look at the two comments in the file.

P.S. I haven’t tried the square of the product of matrices with your ArrayReshape, because I don’t have the latest version where I am right now, I only have version 8.4 here, which doesn’t know about that function. I can try it later today when I get back home to my desktop, but I doubt it will be any different. Maybe Mathematica is not up to this task?

Open in Wolfram Cloud ↗
52

I tried your ArrayReshape. It makes no difference either to the timing for Derivative (way too slow) or the bug with the Re’[1].

I’m a bit surprised that it didn’t make a difference on timing, but I didn’t intend it to fix anything. It’s just more wolfram-idiomatic style.

There is something very wrong with Derivative, and I want you to admit it, instead of putting it on me to come up with something simpler (are you working for Wolfram?).

I’m not a Wolfram employee. I don’t have to admit anything. Finding the simplest demonstration of something is a tried-and-true debugging technique, but if you don’t want to try it, that’s fine by me. I’m not invested in any of this–it’s your baby.

53

What do you find complicated about my example? Maybe it can’t be simplified any further.

54

I need to recreate the lists from the arguments

Is this all you really meant? Inside fn, al is a list of the arguments.

fn[args___] := Module[{al=List[args]},
  ...
]

The arguments can also be given their own names:

On[Assert];
fn[args : PatternSequence[a1_, a2_]] := Module[{al = List[args]},
  Assert[a1 == al[[1]]];
  Assert[a2 == al[[2]]];
  Echo["OK"];
  ]
fn[1, 2]

Inside the 2nd function,

I would like to define, inside f, a list={a1,a2} and then redefine f[a1_,a2_]:=list[[1]]+list[[2]]

What do you mean by “redefine”? The best I can tell is that you’re asking for a functor (a function-rewriting-function) that takes a function in one form, and returns a function that does the same job but in another form. Please provide concrete examples of a function before and after the modification.

If I were to guess, you want to write, literally, a function like: fn[a1_,a2_]:=a1+a2, pass it to a functor, and get back a function like fn[args:PatternSequence[a1_,a2_]]:=Module[{al=List[args]}, al[[1]]+al[[2]] ].

More concretely, you’d want a rewriter that can be used as follows: rewriter[fn[a1_,a2_]:=a1+a2] returns HoldForm[fn[args:PatternSequance[a1_,a2_]]:=Module[{al=List[args]}, al[[1]]+al[[2]] ] ].

The held form can be then released (using ReleaseHold), and the replacement function will be created. The original function can be Clear-ed in the meantime, i.e. after using the rewriter but before the held form is released.

Please confirm that that is indeed what you’d want. Otherwise, provide a concrete example of what you’d pass into the rewriter, and what result you expect back. Thank you.

there is a bug that makes this not work in my case, we don’t have to get into it

That has a strong smell of an XY Problem. Perhaps fixing that “bug” would be desirable instead of devising clumsy workarounds? Could you please give concrete examples of the “bug” you allude to?

I have a much more complicated function in my real case and I want to automate list by using such Mathematica constructs as Table and Array.

There are no fundamental differences between code that generates Part[argList, index] vs code that generates Symbol["arg"<>ToString[index]].

For example, here’s a function that creates a function definition for a function that sums a certain number of arguments. The function definition action (SetDelayed) is inactive, so the form of the function can be inspected before the function definition is performed.

makeSum[fn_Symbol, count_Integer] := 
 Module[{argSymbols, argPatterns},
  argSymbols = Table[Symbol["a" <> ToString[i]], {i, count}];
  argPatterns = 
   Activate[ Inactive[Pattern][#, Blank[]] & /@ argSymbols];
  Inactivate[
   fn[Sequence @@ argPatterns] := Total[argSymbols]
   , SetDelayed
   ]
  ]
Clear[f2]
f2[3, 4] (* evaluates to f2[3,4] since f2 was cleared *)
makeSum[f2, 2] (* returns an inactive function definition *)
Echo[% // Activate, "Activating"] (* now the function definition is performed *)
f2[3, 4] (* evaluates to 7, since f2 is now defined: f2[a1_,a2_]:=a1+a2 *)

Mathematica’s approach to controlling evaluation in the core language is a bit more clumsy than what you’d get in, say LISP macros. There is macro functionality in Mathematica, but it’s not documented, so I didn’t use it, even though it would get rid of some of the wrangling with active and inactive heads.

The argument symbols could have been also shielded from evaluating to something else by temporarily making them have no values (using a Block), but I didn’t bother with that. That’s a TODO.

And of course, what Mathematica version do you use?

A bit of explanation of inactivations folllows. Recall that := is syntactic sugar for SetDelayed. The two lines below are equivalent:

fn[_] := foo
SetDelayed[fn[_], foo]

Thus, we can create an inert function definition that will define the function only once SetDelayed is activated:

Inactivate[ fn[_] := foo, SetDelayed ]
(* equivalent to *)
Inactivate[ SetDelayed[fn[_], foo], SetDelayed ]
(* evaluates to *)
Inactive[SetDelayed][fn[_], foo]

Thus, depending on preference and circumstances, we can either use Inactivate and tell it what heads to deactivate, or use heads wrapped in Inactive directly.

Inactivate[expr_] wraps certain heads inside the expression inside an Inactive head. In Mathematica, expression forms are fully recursive - the head of an expression is usually a symbol, e,g, SetDelayed, but can be an expression itself, e.g. Inactive[SetDelayed]. Example:

Inactivate[5 + 6, Plus]
(* evaluates to *)
Inactive[Plus][5,6]

Then, Activate[%] will reactivate all inactive heads, then the expression will be evaluated, and the addition will be performed, thus Activate[%] here will evaluate to 11.

55

Kuba,
The problem of delaying the evaluation of ffoutsuper till after eqnlist has been generated has been already solved by Eric. The problem of giving ffoutsuper a flat list and reconstructing the actual lists inside of ffoutsuper (so they can be passed on to other functions) has also been already solved by Eric, so thanks but I don’t need that anymore. However, I was wrong to think that this would solve my problem. I thought it would, because when I gave ffoutsuper a flat list and delayed the evaluation of eqnlist with the “Real” hack (imposing that its args be real numbers instead of symbolic or otherwise), everything worked. But now I am thinking that the “Real” hack must have had other effects that made things work (but it doesn’t work with lists, at least as far as Derivative is concerned, which is a BUG. See this thread: Pattern matching doesn't work as expected? and Ginalucca’s response:"
Very curious. It appears that you cannot mix numeric tests like x_Real or x_?NumberQ with list structures inside Derivative")

So the problem in its most general form is to have the same code that worked without lists, work with lists (of different shapes and sizes).

56

Kuba, Eric,
I am including a simple (as simple as I could get it) example to illustrate the bug. It seems to be an inability of NDSolve to handle conditionals. The first cell has a problem (even at t=0) because it can’t deal with Re[Eigenvalues]. The second cell has no problem because I remove the Re[]. This is a bug, but do you have any suggestions for a workaround? I can’t remove the Re in my actual problem, because the matrices are not symmetric and the eigenvalues can become complex (they don’t in this example). Also, in my actual problem I have other conditionals on the equivalent of ff, which might also be causing problems, I’ll check that next. (Note: don’t confuse Re[Eigenvalues] with the Real_ (arguments to ff) hack I was doing initially to delay evaluation of eqnlist. They seem to be unrelated). Not also the nonsense with Re’[4], which is a problem with just Derivative (not even NDSolve) that I showed previously.

Open in Wolfram Cloud ↗
57

*Edited. And here is another bug, which I don’t really understand (I thought problem was Derivative, but I was wrong, forgot to refresh definitions). The first cell is without the conditional, gives good output. Second gives output, but it’s wrong. It has a local variable that gets confused and forgets that it is 1. If you uncomment the commented area, and comment the If statement in ffoutsuper, you will see that it works fine, so the local variable is fine, except if it’s called with the conditional.

Open in Wolfram Cloud ↗
58

I think I solved it! So flattening the lists before passing to ffoutsuper, and reconstructing them should have worked, because that is what I had before, no lists. But I also had the arguments passed to ffoutsuper be _Real. So once I did that, it works again. I don’t really know why. Somehow the conditionals work now. It would have been nice not to have to flatten the lists and reconstruct them. Something for the developers to figure out.