1. Introduction

Pick a memory. For me, perhaps primed by one of Fernández’s examples, it’s a memory of struggling against the waves when I was a kid. I remember flailing around trying to stay afloat. I remember inhaling water and being scraped against the rocks as the waves hit. There was a mild sense of panic and I kept darting my eyes towards the beach. Now, imagine a qualitatively similar situation. Not one from your past but one that’s completely imaginary. The flailing, the inhaling of water, the sense of panic are all the same between these events that are brought to mind. Yet, there seems to be a phenomenal difference between our experience of remembering and our experience of imagining. One way of characterizing this difference is in terms of our temporal phenomenology. Remembering involves a feeling of pastness directed at the remembered event that is missing in imagination. Memory seems to mentally transport us to our personal past. Imagining lacks this feature.

What accounts for this feeling of pastness? In his recent book Memory: a self- referential account (2019), Jordi Fernández develops a comprehensive account of mnemonic content that is supposed to answer this question. Memories, according to Fernández, are self-referential. They represent their own causal origins. According to Fernández, the feeling of pastness that is often associated with episodic memory is grounded in this self-referential causal content rather than any explicit representation of time.¹ In this paper, I will argue that this account falls short and fails to explain how the feeling of pastness can be grounded in the representation of causal origin. The paper goes as follows: Section 2 describes the self-referential theory of episodic memory and how it is supposed to account for the feeling of pastness. This section also lays out Fernández’s arguments against temporal localization views of the feeling of pastness in which the feeling of pastness arises from an explicit representation of the temporal location of remembered events. Section 3 argues that the self-referential explanation of the feeling of pastness falls short. It fails to explain why a representation of cause would give rise to a feeling of pastness without introducing an explicit representation of time. However, if Fernández’s arguments against temporal localization views are correct, then there is no representation of time that could account for the feeling of pastness. Therefore, it would appear that we would have no options available to explain the feeling of pastness. In section 4, I argue that we can avoid this conclusion by appealing to a form of egocentric temporal representation, what I call path-dependent representation, that Fernández does not consider.

2. The self-referential account

Fernández’s book covers an impressive range of topics in the philosophy of memory. The cornerstone of the book is his self-referential account of mnemonic content. I cannot do justice to the many arguments for this position raised in the book. Instead, I will simply present the view and see whether it can do the work he wants it to. Here is the self-referential account of mnemonic content:

For any subject S, memory M and proposition q: If S has M and S would express M by saying that they remember that q, then there is a perceptual experience P that S would express by saying that they perceive that q, such that the content of M is the proposition {W: in W, M is caused by S having perceived that q through P} (^{Fernández, 2019}, p. 78).

Memories, according to the self-referential account, represent themselves as being caused by one of the subject’s perceptual experiences. My memory of struggling against the waves represents itself as having been caused by a perceptual experience of that struggle. The memory will be accurate just in case I really did have a perceptual experience of that struggle and my current memory is a causal effect of that perceptual experience. Contrast that with my imagination of a similar event. The act of imagining that event does not represent itself as being caused by any perceptual episode (even if my capacity to imagine this sort of scenario is partly caused by my having certain experiences).²

How then does this help us account for the feeling of pastness? Let’s begin by getting clear on this phenomenological explanandum. Here is Fernández’s characterization of the feeling of pastness:

(PAST) For any subject S and proposition p: If S remembers that p, then S is aware of the fact that p as obtaining in the past (^{Fernández, 2019}, p. 87).

It’s this awareness that is our phenomenological target. Since explanations of phenomenological features of experience are notoriously problematic, its useful to specify an assumption in the book. Fernández adopts a form of representationalism according to which the phenomenal features of episodic remembering depend on the representational features of episodic memory. The relation is a fairly weak one. As he puts it, “the content of a memory will give us some information about the way in which the memory feels to the subject” (^{Fernández, 2019}, p. 29; emphasis added). For the purposes of this paper, I will simply accept this form of representationalism.

Before seeing how the self-referential view accounts for PAST, it’s useful to see why Fernández thinks that we cannot account for PAST by appealing to what I call temporal localization accounts in which episodic memories represent the temporal location of remembered events. Fernández considers two forms of temporal localization -self- independent and self-dependent accounts. Let’s take these in turn.

SELF-INDEPENDENT TEMPORAL LOCALIZATION. According to this approach, memories locate events in time by placing events at subject-independent temporal locations. My memory of struggling to swim might represent that event as occurring on 15/07/1991. Or perhaps it represents the event as occurring in the early 90s. No reference to the present is made. According to Fernández, this form of temporal localization cannot account for PAST. Consider a case where I imagine what will happen in the future. I might imagine myself struggling to swim and imagine this event as occurring on 15/07/2029. Given that it is currently 2021, the imagined event is in the future and the remembered event is in the past. However, nothing in the content of these mental states captures the pastness of the remembered event or the futureness of the imagined event. Some other content is needed that takes into account my current temporal perspective.

SELF-DEPENDENT TEMPORAL LOCALIZATION. The self-dependent account builds in a temporal perspective into the content that locates remembered events in time. Now, in 2021, when I remember my struggling to swim in 1991, that memory may represent that event as having occurred 30 years ago. Or perhaps it represents the event as having occurred 30-ish years ago, or it just represents the event as having occurred some distance in the past. In all these cases, the memory represents the event as being a certain temporal distance in the past relative to my current temporal perspective. This sort of content would account for PAST, but Fernández argues that this account violates certain intuitions about what counts as an accurate memory.

Consider the following scenario. I struggled to swim in 1991. It’s now 2021 and I’m given the opportunity to travel back in time to 1989. When I travel back I remember that moment where I struggled to swim. It still has its distinctive feeling of pastness. However, according to the self-dependent account of temporal localization, this feeling of pastness is the result of that memory locating the event as being in the past. Is this experience accurate? It seems that a time traveler should still be able to successfully remember their childhood experiences. However, Fernández argues this isn’t possible given the self-dependent form of temporal localization. My memory represents the event as being in the past, but given my time travelling to 1989, the event is not actually in the past but rather it is in the future. The account takes intuitively accurate memories and renders them inaccurate. This is a problem.³

Fernández concludes that we can’t account for PAST through temporal localization. The self-referential account, however, supposedly avoids these worries. Memories represent their causal origin, not the temporal location of the remembered event or perceptual experience. Since in the actual world causes precede their effects, the representation of a memory as being the causal effect of a perceptual experience “goes hand in hand with (that remembered event/perceptual experience) having a certain position in time, namely, being in the past” (^{Fernández, 2019}, p. 108).

It’s at this point that there is a certain ambiguity in Fernández’s account. If memories represent their causal origins, and not the temporal location of remembered events, then were we simply mistaken in describing the distinctive phenomenology of episodic memory as a feeling of pastness? Wouldn’t the resulting phenomenology be one of causal origin since that is what is represented? It would appear that Fernández is giving an error theory of PAST. We actually experience a feeling of causation, but because of the close connection between time and causation, we misdescribe this experience of causation as an experience of pastness. Yet, this isn’t how he describes his position. He takes the self-referential account of memory to ground an awareness of time.

In a later paper, ^{Fernández (2020)} explicitly rejects this error theory in a response to a criticism by Denis ^{Perrin (2018)}. Perrin argues that if Fernández is correct, that memories represent their causal origin and not the pastness of the remembered event, then the pastness of the remembered event must be inferred or derived from this content, and therefore the feeling of pastness is derived from the feeling of causation. But, Perrin continues, this can’t be the case since the feeling of pastness is an intrinsic feature of episodic remembering. The aim here isn’t to assess Perrin’s argument. Rather, it’s Fernández’s response that informs us about his rejection of the error theory and how he understands the relationship between the representation of causal origin and PAST. ^{Fernández (2020)}, agrees with Perrin that the feeling of pastness is not inferred from the content of memory. Instead, he argues that “what happens to the subject, when they have a feeling of pastness associated with their memory, is that they experience one of the things represented by their memory” (^{Fernández, 2020}, p. 294). A little later in the same paragraph, Fernández draws the comparison to the experience of blue. He says,

looking at a blue wall, for example, I experience the fact that my perceptual state represents a certain reflectance property of the surface that I am looking at. I experience it in a qualitatively distinctive way; the way which is characteristic of seeing blue (^{Fernández, 2020}, p. 294).

In the same way that we have genuine experiences of blue by representing reflectance properties, we have genuine experiences of pastness by representing causation.

We have a proposal. The feeling of pastness, a phenomenological feature of episodic memory, is grounded in the representation of causal origin similar to how our experience of blue is grounded in the representation of reflectance properties.

3. Representations of causal and temporal order are distinct

The goal of this section is to show that the self-referential account of PAST falls short. Appealing to the fact that memories represent their causal origins, even if causes as a matter of fact precede their effects, does not explain why memories possess a feeling of pastness.

Recall -the form of representationalism that Fernández adopts- the content of a memory provides information regarding that memory’s phenomenal properties. In the color case, why would the visual representation of reflectance properties provide us with information about the phenomenal blueness that a subject enjoys in virtue of that representational state? We cannot simply appeal to the bare correlation between a state’s having that representational content and its having a certain phenomenal property. That simply restates the explanandum by repeating the fact that states with this content have a certain phenomenology. That isn’t an informative explanation. However, color objectivists (^{Byrne & Hilbert, 2003}; ^{Dretske, 1997}; ^{Tye, 2000}) have an explanation. For something to be represented by our visual system as having a particular reflectance property just is for it be represented as being blue, because the property of blueness just is the property of having a particular type of reflectance property. The informativeness of the account depends on a metaphysical claim about the property picked out by ‘is blue’ and ‘has such and such a reflectance property’.

While there is a debate over the metaphysics of color, the analogous debate doesn’t exist with regards to causation and time.⁴ Causation and time are closely related to one another, yet, they are not identical. Fernández might appeal to a weaker claim. Causal order is not identical to temporal order, but rather, causal order entails temporal order. Call this the temporal priority principle (^{Rankin & McCormack, 2013}) according to which causes precede their effects (either contingently in our world or by necessity). Therefore, memory’s representing causal origin entails something about temporal order, and this entailed content explains the temporal phenomenology. Unfortunately, more would still have to be said about this. In general, we do not seem to have phenomenal experiences of all of the entailments of what our experiences represent. Furthermore, there are empirical reasons for doubting that the metaphysical explanation given by Fernández works. If Fernández is correct, then PAST is explained by the causal content of memory as a result of the metaphysical connection between causation and time. Therefore, if it is the metaphysics that is explaining this phenomenological fact, then if we have other experiences of causation, then we should also find that they will have the corresponding temporal phenomenology. The same metaphysical considerations would hold. However, that is not what we find in the perceptual cases that I will describe below. Our perceptual experiences can violate the temporal priority principle. If the representation of causation in memory explains PAST, then Fernandez will have to appeal to something other than the metaphysics of time and causation to make his point.

Let’s turn our attention to perception to see how experiences of causal and temporal order come apart. It’s widely accepted that we are capable of perceiving causation (^{Michotte, 1955}; ^{Rolfs, Dambacher & Cavanagh, 2013}; ^{Scholl & Tremoulet, 2000}) as well as time (^{Buhusi & Meck, 2005}). It’s also becoming clear that our perception of time and causation influence each other.

Consider how the perceived timing of events serves as a cue for causation. In classic Michotte-style launching experiments, an object will be seen to move across a screen, it will come into contact with another object, and then the second object will begin to move along the same trajectory as the first object. If the objects move smoothly, their trajectories line up in the correct way, and if there is no delay between when the first object comes in contact with the second and the second object’s movement, people will reliably perceive these sequences as launching events in which the first object causes the second to move (Figure 1a). However, if there is a delay between when the objects come into contact and the movement of the second object, then the sense of causation goes away (figure 1b) (^{Michotte, 1955}).

Figure 1 Michotte-style launching displays. (a) subjects perceive this interaction as a causal one when there is no delay; (b) the extended delay between A and B’s movements eliminates the sense of causation. 

The timing of crossmodal cues also influences our perception of causation. In the bounce-stream illusion (^{Sekuler, Sekuler, & Lau, 1997}), subjects are presented with two lines emerging from the top-left and top-right of the screen heading on a diagonal, intersecting in the middle, and continuing on. If only the visual stimulus is presented, subjects will perceive these two lines streaming past one another and continuing on straight paths (Figure 2a). However, if an auditory click is presented when the two streams touch in the middle of the screen, then subjects perceive the two lines as colliding and bouncing off of each other. The result of this apparent causal interaction is that the lines do not proceed along straight paths, but bounce off of each other with the line from the top-left ending up in the bottom-left and vice versa (Figure 2b). Perceived timing in launching experiments and the stream-bounce illusion serve as a cue for the perception of causation.

Figure 2 The bounce-stream illusion. The single and double lines represent perceived trajectories. Actual stimuli did not differ in this way. (a) in the absence of the sound, the lines appear to continue along straight paths. (b) with the properly timed sound the lines appeared to bounce. 

More recent findings show that apparent causal relationships can influence our perception of time. In temporal reordering cases (^{Bechlivanidis & Lagnado, 2013}; 2016), by manipulating causal cues, subjects can be induced to perceive a reversal of temporal order that matches the perceived causal order. In a modified Michotte-Style experiment, three objects, A, B, and C interacted. In the veridical causal condition, object A would move into contact with B, then B would move into contact with C, and then C would move. Subjects reliably perceived the correct causal and temporal order in this condition. The critical condition for their experiment was a modification of this standard sequence in which C would begin to move before B would -i.e. the effect would occur before the cause. In this sequence, subjects reliably misperceived the temporal order of events in a way that respects the temporal priority principle -i.e. they perceived the temporal sequence as A moved, then B, then C. In another condition, the relative timing of B and C was kept the same as the previous condition in which C moved before B, but A was removed from the display. In this case, where the cue for the entire causal sequence is removed, subjects accurately perceived the temporal order of B and C’s movements. Cues regarding causal interaction influenced the perception of temporal order.

Temporal binding effects also show how apparent causal relations can influence temporal perception. In the temporal binding effect, two events at different moments in time are perceived as occurring closer in time than they in fact did provided that the subject has some reason for perceiving one of these events as the cause of the other. A standard experimental design exhibiting this effect involves a subject pressing a button and then a tone is played. The subject will reliably perceive the button press and the tone as being closer in time than they in fact were. However, if the subject is given cues indicating that these events are not causally related, then the temporal binding effect is eliminated (^{Chen & Vroomen, 2013}; ^{Suzuki, Lush, Seth & Roseboom, 2019}).⁵ Causal perception once again seems to influence temporal perception.

Everything said so far is compatible with a representation of causation in memory being “informative” in some sense regarding the experience of time. However, the amenable story ends here. The temporal binding effect can be exploited to produce experiences that violate the temporal priority principle.

In a now classic study by ^{Stetson et al. (2006)}, subjects were asked to press a button and then after a variable delay, with an average length of 35ms, a flash of light would appear on the screen in front of them. Subjects reliably perceived the flash of light as occurring after the button press. In a second block of trials, the experimenters inserted an extended delay of approximately 135ms between the button press and the flash of light. An interesting finding with experimental situations like this is that there is a progressive adaptation effect that leads to a stronger temporal binding effect after repeated trials. The relevant finding came in the third block of trials. After adaptation the extended delay was removed and the flash of light once again appeared approximately 35ms after the button presses. Even though the timing of the stimuli in this third block of trials was identical to the timing from the first block, subjects reliably perceived the flash of light as occurring prior to their button presses. However, recall that the temporal binding effect only took hold given that the subject had a sense of the causal interaction between their button presses and the flash of light. In this study, which has been replicated in (^{Cunningham, Billock, & Tsou, 2001}; ^{Heron et al., 2009}), subjects had a very peculiar perceptual experience. The apparent causal and temporal order of their actions and the effects of those actions violated the temporal priority principle. Effects seemed to occur prior to their causes! In perception, the experience of causal order and the experience of temporal order come apart. A perceptual representation of causal order does not, by itself, ground an experience of pastness or of temporal order. If Fernández wants to account for PAST by appealing to the causal content in memory, then appealing to the metaphysics of time won’t work. The same considerations would apply to perception, yet in perception we see that the temporal priority principle can be violated. If the explanation of PAST developed by Fernández is to succeed, then he must point to something about memory, and not about the metaphysics of time and causation, that would show why his explanation succeeds. Without this additional explanation, we do not have an account of PAST.

However, another option is possible. We take the representation of causation and time to be defeasible cues for one another. The representation of causation in memory could be used as a cue for the representation of temporal location or pastness in memory. An explicit representation of time would provide us with an explanation for our feeling of pastness. Yet, if Fernández is correct, then no version of temporal location can successfully account for PAST. In the next section, I’ll sketch out a version of temporal location that avoids the worries raised by Fernández and allows us to account for PAST.

4. Path dependent representations

The goal of this section is to spell out an alternative subject-dependent account of temporal localization -path-dependent representations. To motivate the account, it will be helpful to begin with spatial representation.

The standard division in the spatial literature is between allocentric and egocentric representations. Allocentric representations pick out spatial locations in a way that makes no necessary reference to an agent. A map can indicate the location of my office in Sheffield without any reference to where I am in relation to my office. This is the spatial analog of Fernández’s subject-independent account of temporal localization.

Egocentric representations, on the other hand, pick out locations relative to an agent. However, types of egocentric representation can differ depending on how they specify locations relative to an agent. Here, I will distinguish between two types of egocentric representation. The first are target representations. Consider my office again. I might represent the office relative to my location by describing a single vector that picks out that location “as the crow flies”. For instance, I might represent my office as being 1.3 miles North of here. This form of egocentric representation is the spatial analog of Fernández’s subject-dependent account of temporal localization.

The other form of egocentric representation are what I call path-dependent representations. Path-dependent representations specify locations by the paths that would be taken to arrive at a target.⁶ This sort of localization should be familiar to someone that has moved to a new city. You might know how to get to your office by following a specific path. I might know that my office is 30 minutes away if you follow this meandering street then turn left at the big intersection, then turn right below the bridge, etc. You can know where your office is in this way without knowing anything about the absolute location of your office or knowing where you stand relative to the office “as the crow flies”.

My suggestion is that we should think of temporal representations of pastness as path-dependent representations. If we do this, then we will have an account of temporal representation that avoids the problems Fernández raised against the temporal location accounts of PAST.

One initial objection to applying path representations to time is that we have multiple dimensions of space that we can travel through. Therefore, different paths in space to a single target can differ in length. However, time is a single dimension. There are no choices of paths between temporal points. As a result, path-dependent representations of time devolve into Fernández’s subject-dependent representations, since our only path through time is “as the crow flies”. However, it’s not so clear that this is true. To see why it is helpful to think about clocks and then we’ll turn to time travel.⁷

Suppose someone travels from London to New York City. They carry a clock while they do this. What does their clock measure? What does the clock of their friend who remained in London measure? In a classical spacetime, the clocks simply measure the absolute amount of time that passes between the person leaving London and arriving in NYC. All observers, and clocks, should agree on how much time passed over this interval. In a relativistic spacetime, like our world, the situation is different. There is no absolute time. Instead, our clocks measure the temporal distance along the distinct spacetime worldlines traced by the traveler and their sedentary friend. As a result, the two clocks will disagree on how much time has passed since relativistic effects change the temporal length of these worldlines. This isn’t just a theoretical result. It’s something that has been observed and measured.⁸ A regular stopwatch already measures the length of a temporal path and not absolute time.

Consider our time travel case again. I remember struggling to swim at some point in my past. That event took place in 1991. It is now 2021. I step into a time machine and emerge in 1989. I still remember this event accurately and still have a feeling of pastness associated with my memory. How do we account for this feeling of pastness? It’s useful to distinguish between two notions of time in these scenarios. There is external time and personal time (^{Lewis, 1976}). External time locates events in time via an egocentric target framework. In 1989, the remembered event is two years in the future. However, personal time is the time measured by the time traveler’s watch - it’s a measure of time along my personal worldline. As I enter my time machine and travel backwards in (external) time, from my perspective time and causation continue to move forward.

When I remember my struggling to swim in my past, I do not represent the pastness of this event in external time. Rather, I represent the pastness of the event in a path- dependent way -it is my personal causally continuous past. When I emerge in 1989, the remembered event will be in the past of my worldline. It will be represented as being in that temporal past that was causally relevant to my current memory (perhaps through a memory trace). There is no inaccuracy here. We have a way of temporally locating events that seems to capture PAST. Given the representationalism that Fernández endorses, the fact that our memories represent events as being in our path-dependent past would explain why memories have the phenomenology of pastness.

The suggestion here isn’t that we use (internal) clocks to represent the temporal location of remembered events. Rather, we use a variety of cues to locate remembered events in a path-dependent representation of time. The causal content of memory could be a cue for this sort of representation. That a perceptual episode caused our current memory is reason for representing that that perceptual episode was in the (path-dependent) past since the causal influence from that perceptual episode moves forward along our worldline.

Fernández could maintain the vast majority of his self-referential account and still provide an explanation of PAST if he includes path-dependent temporal content as a component of his theory. Otherwise, he would have to either abandon attempts at explaining PAST, since causal content alone does not suffice, or he has to provide a reason for thinking that the representation of causal order in memory has a different connection to time and temporal phenomenology than it does in perception.

However, this account of temporal representation is also compatible with constructivist approaches to memory (^{De Brigard, 2014}; ^{Michaelian, 2016}). If, in the act of recollection, we utilize a variety of mental representations to construct a representation of a past event, then path-dependent temporal representations may be an element in this process. Information about causal origin, along with other information, may serve as a cue for forming the appropriate path-dependent temporal representation in the act of recollection.

Finally, I’m not suggesting that in remembering we have a theoretical grasp of path-dependent temporal representation. Rather, we utilize path-dependent representations of time to locate events. If we accept anti-individualism about content (^{Burge, 1979}), i.e. that what we represent is determined by the world, and we accept that temporal properties are path-dependent, then our temporal representations will latch onto path-dependent temporal properties. In the same way that we can use clocks that measure path-dependent time without understanding relativistic physics, the same is true for the mental representation of time.

5. Conclusion

The target of this paper was whether the self-referential theory could account for PAST. I gave reasons for why it can’t. It can, however, be augmented with path- dependent temporal representations while retaining the connection between causation and temporal order that Fernández was appealing to. The difference is that it involves explicit temporal representation.

An important question has been lingering beneath the surface of this paper. Memories are complex and what information is accessed at the moment of recall often depends on the context and purpose of recall (^{Michaelian, 2016}). Much of this information is informative regarding the timing of events. Things like familiarity, visual cues, location, etc. can be used to infer something about time. Phenomenology is also difficult to report. It may be the case that the distinctive phenomenology of memory, what we described as PAST, might not be a single phenomenal experience. Rather, remembering may have various phenomenal features, some temporal, some causal, some familiarity-based (^{Perrin, Michaelian & Sant’Anna, 2020}), and we simply describe these phenomenal features in temporal terms due to our familiarity with temporal discourse. This paper hasn’t engaged with this question. However, if we want to account for PAST, something more than representations of causation is needed. I’ve articulated an account of path-dependent temporal representations that can help.