✨
This commit is contained in:
Binary file not shown.
@ -59,51 +59,50 @@
|
|||||||
|
|
||||||
Finally, extending MEOW's event-based scheduler to support network event triggers would enable the simple and efficient exchange of data between workflows running on different machines. This feature is particularly valuable in distributed computing environments, where data processing tasks are often split across multiple systems to maximize resource utilization and minimize latency.
|
Finally, extending MEOW's event-based scheduler to support network event triggers would enable the simple and efficient exchange of data between workflows running on different machines. This feature is particularly valuable in distributed computing environments, where data processing tasks are often split across multiple systems to maximize resource utilization and minimize latency.
|
||||||
|
|
||||||
\begin{tcolorbox}[colback=lightgray!30!white]
|
Integrating network event triggers into MEOW would provide an advantage specifically in the context of heterogeneous workflows, which incorporate a mix of different tasks running on diverse computing environments. By their nature, these workflows can involve tasks running on different systems, potentially even in different physical locations, which need to exchange data or coordinate their progress. Currently, MEOW's reliance on local file events as triggers can be a limiting factor in these scenarios. Network event triggers offer a powerful solution to this challenge. They can not only handle tasks running across different machines, but also dynamically adapt to the changing requirements of a heterogeneous workflow, such as triggering new tasks based on the results of remote computations. Thus, the addition of network event triggers is a significant step in enhancing MEOW's already robust handling of heterogeneous workflows, bolstering its utility in today's diverse and distributed computing landscape.
|
||||||
How exactly would network events improve things? Expand using graph David sent to me.
|
|
||||||
\end{tcolorbox}
|
|
||||||
|
|
||||||
\begin{tcolorbox}[colback=lightgray!30!white]
|
|
||||||
Move this to after method (Future work).
|
|
||||||
\end{tcolorbox}
|
|
||||||
One specific example of a use-case where network event triggers could prove useful is the workflow for The Brain Imaging Data Structure (BIDS). The BIDS workflow requires data to be sent between multiple machines and validated by a user. Network event triggers could streamline this process by automatically initiating data transfer tasks when specific conditions are met, thereby reducing the need for manual management. Additionally, network triggers could facilitate user validation by allowing users to manually prompt the continuation of the workflow through specific network requests, simplifying the user's role in the validation process.
|
|
||||||
|
|
||||||
\begin{figure}[H]
|
\begin{figure}[H]
|
||||||
\begin{center}
|
\begin{center}
|
||||||
\includegraphics[width=0.6\textwidth]{src/BIDS.png}
|
\includegraphics[width=\textwidth]{src/heterogeneous.png}
|
||||||
\end{center}
|
\end{center}
|
||||||
\caption{\textbf{Temp graph. Replace.} The structure of the BIDS workflow.}
|
\caption{\textbf{Temp.} An example of a heterogeneous workflow}
|
||||||
\end{figure}
|
\end{figure}
|
||||||
|
|
||||||
\subsection{Background}
|
\subsection{Background}
|
||||||
\subsubsection{The structure of MEOW}
|
\subsubsection{The structure of MEOW}
|
||||||
|
|
||||||
|
The MEOW event-based scheduler consists of four main components: \textit{monitors}, \textit{handlers}, \textit{the conductor}, and \textit{the runner}.
|
||||||
|
|
||||||
|
Monitors listen for triggering events. They are initialized with a number of \textit{patterns}, which describe the triggering event. When a pattern's triggering event occurs, the monitor signals to the conductor that the pattern has been triggered, and schedules a job that has been associated with the pattern.
|
||||||
|
|
||||||
|
\begin{figure}[H]
|
||||||
|
\begin{center}
|
||||||
|
\includegraphics[width=0.6\textwidth]{src/monitor.png}
|
||||||
|
\end{center}
|
||||||
|
\caption{The monitor's role in MEOW's event-based system.}
|
||||||
|
\end{figure}
|
||||||
|
|
||||||
\begin{tcolorbox}[colback=lightgray!30!white]
|
\begin{tcolorbox}[colback=lightgray!30!white]
|
||||||
Expand. More diagrams.
|
I haven't used "Resources" to describe the job queue. Should I do that or should I rephrase the diagram to be more in line with the rest of the project?
|
||||||
\end{tcolorbox}
|
\end{tcolorbox}
|
||||||
|
|
||||||
The MEOW event-based scheduler has three main parts: \textit{monitors}, \textit{handlers}, and \textit{the conductor}.
|
Handlers perform actions and jobs on behalf of the scheduler. They are initialized with a number of \textit{recipes}, which describe the action to be taken. The handler starts a job when signal to do so by the conductor.
|
||||||
|
|
||||||
|
The conductor handles the jobs queue. It is initialized with a number of rules, which a pattern paired with a recipe. When a monitor sends it a triggered pattern, the rules are checked for that pattern. If one or more rules contain that pattern, the corresponding recipes are triggered in their handler.
|
||||||
|
|
||||||
|
Finally, the runner is the main program that orchestrates all these components. Each instance of the runner incorporates at least one instance of a monitor, handler, and conductor.
|
||||||
|
|
||||||
\begin{figure}[H]
|
\begin{figure}[H]
|
||||||
\begin{center}
|
\begin{center}
|
||||||
\begin{tikzpicture}
|
\begin{tikzpicture}
|
||||||
|
\node[draw,rectangle,rounded corners,text width=8cm,align=center] at (0,2) (run) {Runner};
|
||||||
\node[draw,rectangle,rounded corners] at (0,0) (con) {Conductor};
|
\node[draw,rectangle,rounded corners] at (0,0) (con) {Conductor};
|
||||||
\node[draw,rectangle,rounded corners] at (3,-2) (mon) {Monitor};
|
\node[draw,rectangle,rounded corners] at (3,-2) (mon) {Monitor};
|
||||||
\node[draw,rectangle,rounded corners] at (-3,-2) (han) {Handler};
|
\node[draw,rectangle,rounded corners] at (-3,-2) (han) {Handler};
|
||||||
\end{tikzpicture}
|
\end{tikzpicture}
|
||||||
\end{center}
|
\end{center}
|
||||||
\caption{\textbf{WIP.} How the three elements of MEOW interact.}
|
\caption{\textbf{WIP.} How the elements of MEOW interact.}
|
||||||
\end{figure}
|
\end{figure}
|
||||||
\begin{tcolorbox}[colback=lightgray!30!white]
|
|
||||||
Include the runner in diagram.
|
|
||||||
\end{tcolorbox}
|
|
||||||
|
|
||||||
|
|
||||||
Monitors listen for triggering events. They are initialized with a number of \textit{patterns}, which describe the triggering event. When a pattern's triggering event occurs, the monitor signals to the conductor that the pattern has been triggered.
|
|
||||||
|
|
||||||
Handlers perform actions and jobs on behalf of the scheduler. They are initialized with a number of \textit{recipes}, which describe the action to be taken. The handler starts a job when signal to do so by the conductor.
|
|
||||||
|
|
||||||
The conductor handles the jobs queue. It is initialized with a number of rules, which a pattern paired with a recipe. When a monitor sends it a triggered pattern, the rules are checked for that pattern. If one or more rules contain that pattern, the corresponding recipes are triggered in their handler.
|
|
||||||
|
|
||||||
\subsubsection{The \texttt{meow\_base} codebase}
|
\subsubsection{The \texttt{meow\_base} codebase}
|
||||||
\begin{tcolorbox}[colback=lightgray!30!white]
|
\begin{tcolorbox}[colback=lightgray!30!white]
|
||||||
@ -170,6 +169,18 @@
|
|||||||
\end{itemize}
|
\end{itemize}
|
||||||
\end{tcolorbox}
|
\end{tcolorbox}
|
||||||
|
|
||||||
|
\begin{tcolorbox}[colback=lightgray!30!white]
|
||||||
|
Give context to following paragraph.
|
||||||
|
\end{tcolorbox}
|
||||||
|
One specific example of a use-case where network event triggers could prove useful is the workflow for The Brain Imaging Data Structure (BIDS). The BIDS workflow requires data to be sent between multiple machines and validated by a user. Network event triggers could streamline this process by automatically initiating data transfer tasks when specific conditions are met, thereby reducing the need for manual management. Additionally, network triggers could facilitate user validation by allowing users to manually prompt the continuation of the workflow through specific network requests, simplifying the user's role in the validation process.
|
||||||
|
|
||||||
|
\begin{figure}[H]
|
||||||
|
\begin{center}
|
||||||
|
\includegraphics[width=0.6\textwidth]{src/BIDS.png}
|
||||||
|
\end{center}
|
||||||
|
\caption{\textbf{Temp graph. Replace.} The structure of the BIDS workflow. Data is transferred to user, and to the cloud.}
|
||||||
|
\end{figure}
|
||||||
|
|
||||||
\section{Conclusion}
|
\section{Conclusion}
|
||||||
\begin{tcolorbox}[colback=lightgray!30!white]
|
\begin{tcolorbox}[colback=lightgray!30!white]
|
||||||
Did I succeed in what I wanted to do?
|
Did I succeed in what I wanted to do?
|
||||||
|
BIN
src/heterogeneous.png
Normal file
BIN
src/heterogeneous.png
Normal file
Binary file not shown.
After Width: | Height: | Size: 139 KiB |
BIN
src/monitor.png
Normal file
BIN
src/monitor.png
Normal file
Binary file not shown.
After Width: | Height: | Size: 16 KiB |
Reference in New Issue
Block a user