For general server architecture, see README.md. This module implements the watchdog process.

Watchdog state

Most LSP clients only send us file diffs, so to facilitate sending entire file contents to freshly restarted workers, the watchdog needs to maintain the current state of each file. It can also use this state to detect changes to the header and thus restart the corresponding worker, freeing its imports.

TODO(WN): We may eventually want to keep track of approximately (since this isn't knowable exactly) where in the file a worker crashed. Then on restart, we tell said worker to only parse up to that point and query the user about how to proceed (continue OR allow the user to fix the bug and then continue OR ..). Without this, if the crash is deterministic, users may be confused about why the server seemingly stopped working for a single file.

Watchdog <-> worker communication

The watchdog process and its file worker processes communicate via LSP. If the necessity arises, we might add non-standard commands similarly based on JSON-RPC. Most requests and notifications are forwarded to the corresponding file worker process, with the exception of these notifications:

• textDocument/didOpen: Launch the file worker, create the associated watchdog state and launch a task to asynchronously receive LSP packets from the worker (e.g. request responses).
• textDocument/didChange: Update the local file state so that it can be resent to restarted workers. Then forward the didChange notification.
• textDocument/didClose: Signal a shutdown to the file worker and remove the associated watchdog state.

Moreover, we don't implement the full protocol at this level:

• Upon starting, the initialize request is forwarded to the worker, but it must not respond with its server capabilities. Consequently, the watchdog will not send an initialized notification to the worker.
• After initialize, the watchdog sends the corresponding didOpen notification with the full current state of the file. No additional didOpen notifications will be forwarded to the worker process.
• $/cancelRequest notifications are forwarded to all file workers.
• File workers are always terminated with an exit notification, without previously receiving a shutdown request. Similarly, they never receive a didClose notification.

Watchdog <-> client communication

The watchdog itself should implement the LSP standard as closely as possible. However we reserve the right to add non-standard extensions in case they're needed, for example to communicate tactic state.

def Lean.Server.Watchdog.workerCfg : IO.Process.StdioConfig

Equations

• Lean.Server.Watchdog.workerCfg = { stdin := IO.Process.Stdio.piped, stdout := IO.Process.Stdio.piped, stderr := IO.Process.Stdio.inherit }

inductive Lean.Server.Watchdog.WorkerEvent : Type

Events that worker-specific tasks signal to the main thread.

• terminated: Lean.Server.Watchdog.WorkerEvent
• importsChanged: Lean.Server.Watchdog.WorkerEvent
• crashed: IO.Error → Lean.Server.Watchdog.WorkerEvent
• ioError: IO.Error → Lean.Server.Watchdog.WorkerEvent

inductive Lean.Server.Watchdog.CrashOrigin : Type

• fileWorkerToClientForwarding: Lean.Server.Watchdog.CrashOrigin
• clientToFileWorkerForwarding: Lean.Server.Watchdog.CrashOrigin

inductive Lean.Server.Watchdog.WorkerState : Type

• crashed: Array Lean.JsonRpc.Message → Lean.Server.Watchdog.CrashOrigin → Lean.Server.Watchdog.WorkerState

  The watchdog can detect a crashed file worker in two places: When trying to send a message to the file worker and when reading a request reply. In the latter case, the forwarding task terminates and delegates a crashed event to the main task. Then, in both cases, the file worker has its state set to crashed and requests that are in-flight are errored. Upon receiving the next packet for that file worker, the file worker is restarted and the packet is forwarded to it. If the crash was detected while writing a packet, we queue that packet until the next packet for the file worker arrives.

• running: Lean.Server.Watchdog.WorkerState

@[reducible, inline]

abbrev Lean.Server.Watchdog.PendingRequestMap : Type

Equations

• Lean.Server.Watchdog.PendingRequestMap = Lean.RBMap Lean.JsonRpc.RequestID Lean.JsonRpc.Message compare

structure Lean.Server.Watchdog.FileWorker : Type

• doc : Lean.Server.DocumentMeta
• proc : IO.Process.Child Lean.Server.Watchdog.workerCfg
• commTask : Task Lean.Server.Watchdog.WorkerEvent
• state : Lean.Server.Watchdog.WorkerState
• pendingRequestsRef : IO.Ref Lean.Server.Watchdog.PendingRequestMap

  The pending requests map contains all requests that have been received from the LSP client, but were not answered yet. We need them for forwarding cancellation requests to the correct worker as well as cleanly aborting requests on worker crashes.

def Lean.Server.Watchdog.FileWorker.stdin (fw : Lean.Server.Watchdog.FileWorker) : IO.FS.Stream

Equations

• fw.stdin = IO.FS.Stream.ofHandle fw.proc.stdin

def Lean.Server.Watchdog.FileWorker.stdout (fw : Lean.Server.Watchdog.FileWorker) : IO.FS.Stream

Equations

• fw.stdout = IO.FS.Stream.ofHandle fw.proc.stdout

def Lean.Server.Watchdog.FileWorker.erasePendingRequest (fw : Lean.Server.Watchdog.FileWorker) (id : Lean.JsonRpc.RequestID) : IO Unit

Equations

• fw.erasePendingRequest id = ST.Ref.modify fw.pendingRequestsRef fun (pendingRequests : Lean.Server.Watchdog.PendingRequestMap) => Lean.RBMap.erase pendingRequests id

def Lean.Server.Watchdog.FileWorker.errorPendingRequests (fw : Lean.Server.Watchdog.FileWorker) (hError : IO.FS.Stream) (code : Lean.JsonRpc.ErrorCode) (msg : String) : IO Unit

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.FileWorker.queuedMsgs (fw : Lean.Server.Watchdog.FileWorker) : Array Lean.JsonRpc.Message

Equations

• fw.queuedMsgs = match fw.state with | Lean.Server.Watchdog.WorkerState.running => #[] | Lean.Server.Watchdog.WorkerState.crashed queuedMsgs origin => queuedMsgs

@[reducible, inline]

abbrev Lean.Server.Watchdog.FileWorkerMap : Type

Equations

• Lean.Server.Watchdog.FileWorkerMap = Lean.RBMap Lean.Lsp.DocumentUri Lean.Server.Watchdog.FileWorker compare

@[reducible, inline]

abbrev Lean.Server.Watchdog.ImportMap : Type

Equations

• Lean.Server.Watchdog.ImportMap = Lean.RBMap Lean.Lsp.DocumentUri (Lean.RBTree Lean.Lsp.DocumentUri compare) compare

structure Lean.Server.Watchdog.ImportData : Type

Global import data for all open files managed by this watchdog.

• imports : Lean.Server.Watchdog.ImportMap

  For every open file, the files that it imports.

• importedBy : Lean.Server.Watchdog.ImportMap

  For every open file, the files that it is imported by.

def Lean.Server.Watchdog.ImportData.update (d : Lean.Server.Watchdog.ImportData) (uri : Lean.Lsp.DocumentUri) (imports : Lean.RBTree Lean.Lsp.DocumentUri compare) : Lean.Server.Watchdog.ImportData

Updates d with the new set of imports for the file uri.

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.ImportData.eraseImportsOf (d : Lean.Server.Watchdog.ImportData) (uri : Lean.Lsp.DocumentUri) : Lean.Server.Watchdog.ImportData

Sets the imports of uri in d to the empty set.

Equations

• d.eraseImportsOf uri = d.update uri ∅

structure Lean.Server.Watchdog.RequestIDTranslation : Type

• sourceUri : Lean.Lsp.DocumentUri
• localID : Lean.JsonRpc.RequestID

@[reducible, inline]

abbrev Lean.Server.Watchdog.PendingServerRequestMap : Type

Equations

• Lean.Server.Watchdog.PendingServerRequestMap = Lean.RBMap Lean.JsonRpc.RequestID Lean.Server.Watchdog.RequestIDTranslation compare

structure Lean.Server.Watchdog.ServerRequestData : Type

• pendingServerRequests : Lean.Server.Watchdog.PendingServerRequestMap
• freshServerRequestID : Nat

def Lean.Server.Watchdog.ServerRequestData.trackOutboundRequest (data : Lean.Server.Watchdog.ServerRequestData) (sourceUri : Lean.Lsp.DocumentUri) (localID : Lean.JsonRpc.RequestID) : Lean.JsonRpc.RequestID × Lean.Server.Watchdog.ServerRequestData

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.ServerRequestData.translateInboundResponse (data : Lean.Server.Watchdog.ServerRequestData) (globalID : Lean.JsonRpc.RequestID) : Option Lean.Server.Watchdog.RequestIDTranslation × Lean.Server.Watchdog.ServerRequestData

Equations

• One or more equations did not get rendered due to their size.

structure Lean.Server.Watchdog.ServerContext : Type

• hIn : IO.FS.Stream
• hOut : IO.FS.Stream
• hLog : IO.FS.Stream
• args : List String

  Command line arguments.

• fileWorkersRef : IO.Ref Lean.Server.Watchdog.FileWorkerMap
• initParams : Lean.Lsp.InitializeParams

  We store these to pass them to workers.

• workerPath : Lake.FilePath
• srcSearchPath : Lake.SearchPath
• references : IO.Ref Lean.Server.References
• serverRequestData : IO.Ref Lean.Server.Watchdog.ServerRequestData
• importData : IO.Ref Lean.Server.Watchdog.ImportData

@[reducible, inline]

abbrev Lean.Server.Watchdog.ServerM (α : Type) : Type

Equations

• Lean.Server.Watchdog.ServerM = ReaderT Lean.Server.Watchdog.ServerContext IO

def Lean.Server.Watchdog.updateFileWorkers (val : Lean.Server.Watchdog.FileWorker) : Lean.Server.Watchdog.ServerM Unit

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.findFileWorker? (uri : Lean.Lsp.DocumentUri) : Lean.Server.Watchdog.ServerM (Option Lean.Server.Watchdog.FileWorker)

Equations

• Lean.Server.Watchdog.findFileWorker? uri = do let __do_lift ← read let __do_lift ← ST.Ref.get __do_lift.fileWorkersRef pure (Lean.RBMap.find? __do_lift uri)

def Lean.Server.Watchdog.eraseFileWorker (uri : Lean.Lsp.DocumentUri) : Lean.Server.Watchdog.ServerM Unit

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.log (msg : String) : Lean.Server.Watchdog.ServerM Unit

Equations

• Lean.Server.Watchdog.log msg = do let st ← read liftM (st.hLog.putStrLn msg) liftM st.hLog.flush

def Lean.Server.Watchdog.handleIleanInfoUpdate (fw : Lean.Server.Watchdog.FileWorker) (params : Lean.Lsp.LeanIleanInfoParams) : Lean.Server.Watchdog.ServerM Unit

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.handleIleanInfoFinal (fw : Lean.Server.Watchdog.FileWorker) (params : Lean.Lsp.LeanIleanInfoParams) : Lean.Server.Watchdog.ServerM Unit

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.handleImportClosure (fw : Lean.Server.Watchdog.FileWorker) (params : Lean.Lsp.LeanImportClosureParams) : Lean.Server.Watchdog.ServerM Unit

Updates the global import data with the import closure provided by the file worker after it successfully processed its header.

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.startFileWorker (m : Lean.Server.DocumentMeta) : Lean.Server.Watchdog.ServerM Unit

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.terminateFileWorker (uri : Lean.Lsp.DocumentUri) : Lean.Server.Watchdog.ServerM Unit

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.handleCrash (uri : Lean.Lsp.DocumentUri) (queuedMsgs : Array Lean.JsonRpc.Message) (origin : Lean.Server.Watchdog.CrashOrigin) : Lean.Server.Watchdog.ServerM Unit

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.tryDischargeQueuedMessages (uri : Lean.Lsp.DocumentUri) (queuedMsgs : Array Lean.JsonRpc.Message) : Lean.Server.Watchdog.ServerM Unit

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.tryWriteMessage (uri : Lean.Lsp.DocumentUri) (msg : Lean.JsonRpc.Message) (queueFailedMessage : optParam Bool true) (restartCrashedWorker : optParam Bool false) : Lean.Server.Watchdog.ServerM Unit

Tries to write a message, sets the state of the FileWorker to crashed if it does not succeed and restarts the file worker if the crashed flag was already set. Just logs an error if there is no FileWorker at this uri. Messages that couldn't be sent can be queued up via the queueFailedMessage flag and will be discharged after the FileWorker is restarted.

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.notifyAboutStaleDependency (uri : Lean.Lsp.DocumentUri) (staleDependency : Lean.Lsp.DocumentUri) : Lean.Server.Watchdog.ServerM Unit

Sends a notification to the file worker identified by uri that its dependency staleDependency is out-of-date.

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.findDefinitions (p : Lean.Lsp.TextDocumentPositionParams) : Lean.Server.Watchdog.ServerM (Array Lean.Lsp.Location)

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.handleReference (p : Lean.Lsp.ReferenceParams) : Lean.Server.Watchdog.ServerM (Array Lean.Lsp.Location)

Equations

• One or more equations did not get rendered due to their size.

structure Lean.Server.Watchdog.CallHierarchyItemData : Type

Used in CallHierarchyItem.data? to retain all the data needed to quickly re-identify the call hierarchy item.

• module : Lake.Name
• name : Lake.Name

instance Lean.Server.Watchdog.instFromJsonCallHierarchyItemData : Lean.FromJson Lean.Server.Watchdog.CallHierarchyItemData

Equations

• Lean.Server.Watchdog.instFromJsonCallHierarchyItemData = { fromJson? := Lean.Server.Watchdog.fromJsonCallHierarchyItemData✝ }

instance Lean.Server.Watchdog.instToJsonCallHierarchyItemData : Lean.ToJson Lean.Server.Watchdog.CallHierarchyItemData

Equations

• Lean.Server.Watchdog.instToJsonCallHierarchyItemData = { toJson := Lean.Server.Watchdog.toJsonCallHierarchyItemData✝ }

def Lean.Server.Watchdog.CallHierarchyItemData.fromItem? (item : Lean.Lsp.CallHierarchyItem) : Option Lean.Server.Watchdog.CallHierarchyItemData

Extracts the CallHierarchyItemData from item.data? and returns none if this is not possible.

Equations

• Lean.Server.Watchdog.CallHierarchyItemData.fromItem? item = do let __do_lift ← item.data? (Lean.fromJson? __do_lift).toOption

def Lean.Server.Watchdog.handlePrepareCallHierarchy (p : Lean.Lsp.CallHierarchyPrepareParams) : Lean.Server.Watchdog.ServerM (Array Lean.Lsp.CallHierarchyItem)

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.handleCallHierarchyIncomingCalls (p : Lean.Lsp.CallHierarchyIncomingCallsParams) : Lean.Server.Watchdog.ServerM (Array Lean.Lsp.CallHierarchyIncomingCall)

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.handleCallHierarchyIncomingCalls.collapseSameIncomingCalls (incomingCalls : Array Lean.Lsp.CallHierarchyIncomingCall) : Array Lean.Lsp.CallHierarchyIncomingCall

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.handleCallHierarchyOutgoingCalls (p : Lean.Lsp.CallHierarchyOutgoingCallsParams) : Lean.Server.Watchdog.ServerM (Array Lean.Lsp.CallHierarchyOutgoingCall)

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.handleCallHierarchyOutgoingCalls.collapseSameOutgoingCalls (outgoingCalls : Array Lean.Lsp.CallHierarchyOutgoingCall) : Array Lean.Lsp.CallHierarchyOutgoingCall

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.handleWorkspaceSymbol (p : Lean.Lsp.WorkspaceSymbolParams) : Lean.Server.Watchdog.ServerM (Array Lean.Lsp.SymbolInformation)

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.handlePrepareRename (p : Lean.Lsp.PrepareRenameParams) : Lean.Server.Watchdog.ServerM (Option Lean.Lsp.Range)

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.handleRename (p : Lean.Lsp.RenameParams) : Lean.Server.Watchdog.ServerM Lean.Lsp.WorkspaceEdit

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.handleDidOpen (p : Lean.Lsp.LeanDidOpenTextDocumentParams) : Lean.Server.Watchdog.ServerM Unit

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.handleDidChange (p : Lean.Lsp.DidChangeTextDocumentParams) : Lean.Server.Watchdog.ServerM Unit

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.handleDidSave (p : Lean.Lsp.DidSaveTextDocumentParams) : Lean.Server.Watchdog.ServerM Unit

When a file is saved, notifies all file workers for files that depend on this file that this specific import is now stale so that the file worker can issue a diagnostic asking users to restart the file.

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.handleDidClose (p : Lean.Lsp.DidCloseTextDocumentParams) : Lean.Server.Watchdog.ServerM Unit

Equations

• Lean.Server.Watchdog.handleDidClose p = Lean.Server.Watchdog.terminateFileWorker p.textDocument.uri

def Lean.Server.Watchdog.handleDidChangeWatchedFiles (p : Lean.Lsp.DidChangeWatchedFilesParams) : Lean.Server.Watchdog.ServerM Unit

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.handleCancelRequest (p : Lean.Lsp.CancelParams) : Lean.Server.Watchdog.ServerM Unit

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.forwardNotification {α : Type} [Lean.ToJson α] [Lean.Lsp.FileSource α] (method : String) (params : α) : Lean.Server.Watchdog.ServerM Unit

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.parseParams (paramType : Type) [Lean.FromJson paramType] (params : Lean.Json) : Lean.Server.Watchdog.ServerM paramType

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.forwardRequestToWorker (id : Lean.JsonRpc.RequestID) (method : String) (params : Lean.Json) : Lean.Server.Watchdog.ServerM Unit

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.handleRequest (id : Lean.JsonRpc.RequestID) (method : String) (params : Lean.Json) : Lean.Server.Watchdog.ServerM Unit

def Lean.Server.Watchdog.handleNotification (method : String) (params : Lean.Json) : Lean.Server.Watchdog.ServerM Unit

def Lean.Server.Watchdog.handleResponse (id : Lean.JsonRpc.RequestID) (result : Lean.Json) : Lean.Server.Watchdog.ServerM Unit

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.handleResponseError (id : Lean.JsonRpc.RequestID) (code : Lean.JsonRpc.ErrorCode) (message : String) (data? : Option Lean.Json) : Lean.Server.Watchdog.ServerM Unit

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.shutdown : Lean.Server.Watchdog.ServerM Unit

Equations

• One or more equations did not get rendered due to their size.

inductive Lean.Server.Watchdog.ServerEvent : Type

• workerEvent: Lean.Server.Watchdog.FileWorker → Lean.Server.Watchdog.WorkerEvent → Lean.Server.Watchdog.ServerEvent
• clientMsg: Lean.JsonRpc.Message → Lean.Server.Watchdog.ServerEvent
• clientError: IO.Error → Lean.Server.Watchdog.ServerEvent

def Lean.Server.Watchdog.runClientTask : Lean.Server.Watchdog.ServerM (Task Lean.Server.Watchdog.ServerEvent)

Equations

• One or more equations did not get rendered due to their size.

partial def Lean.Server.Watchdog.mainLoop (clientTask : Task Lean.Server.Watchdog.ServerEvent) : Lean.Server.Watchdog.ServerM Unit

def Lean.Server.Watchdog.mkLeanServerCapabilities : Lean.Lsp.ServerCapabilities

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.initAndRunWatchdogAux : Lean.Server.Watchdog.ServerM Unit

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.findWorkerPath : IO Lake.FilePath

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.startLoadingReferences (references : IO.Ref Lean.Server.References) : IO Unit

Starts loading .ileans present in the search path asynchronously in an IO task. This ensures that server startup is not blocked by loading the .ileans. In return, while the .ileans are being loaded, users will only get incomplete results in requests that need references.

Equations

• One or more equations did not get rendered due to their size.

def Lean.Server.Watchdog.initAndRunWatchdog (args : List String) (i : IO.FS.Stream) (o : IO.FS.Stream) (e : IO.FS.Stream) : IO Unit

Equations

• One or more equations did not get rendered due to their size.

@[export lean_server_watchdog_main]

def Lean.Server.Watchdog.watchdogMain (args : List String) : IO UInt32

Equations

• One or more equations did not get rendered due to their size.