✨
This commit is contained in:
5
.vscode/settings.json
vendored
Normal file
5
.vscode/settings.json
vendored
Normal file
@ -0,0 +1,5 @@
|
||||
{
|
||||
"cSpell.words": [
|
||||
"SWMSs"
|
||||
]
|
||||
}
|
Binary file not shown.
@ -1,7 +1,10 @@
|
||||
\documentclass[a4paper]{article}
|
||||
\documentclass[a4paper,twocolumn]{article}
|
||||
\usepackage[margin=1in]{geometry}
|
||||
\usepackage[most]{tcolorbox}
|
||||
\usepackage{xcolor}
|
||||
\usepackage{fancyhdr} % for headers
|
||||
\usepackage[citestyle=verbose-ibid, backend=biber, autocite=footnote]{biblatex} % Footnote references. Use autocite{}.
|
||||
% \usepackage[citestyle=verbose-ibid, backend=biber, autocite=footnote]{biblatex} % Footnote references. Use autocite{}.
|
||||
\usepackage{biblatex}
|
||||
|
||||
|
||||
% --- Configuration ---
|
||||
@ -15,68 +18,88 @@
|
||||
% \rhead{\thepage}
|
||||
% \fancyheadoffset{0.005\textwidth}
|
||||
|
||||
\setlength{\parskip}{5pt}
|
||||
|
||||
\begin{document}
|
||||
\section{Abstract}
|
||||
\textit{Explains briefly the paper and what it does.}
|
||||
This paper...
|
||||
\begin{tcolorbox}[colback=lightgray!30!white]
|
||||
Explain briefly the paper and what it does.
|
||||
\end{tcolorbox}
|
||||
|
||||
\section{Introduction}
|
||||
\textit{Introduce the scope.}
|
||||
|
||||
\begin{itemize}
|
||||
\item Event-based scheduling as a concept \begin{itemize}
|
||||
\item Why is it useful?
|
||||
\end{itemize}
|
||||
\item Network events and REST APIs as a concept \begin{itemize}
|
||||
\item
|
||||
\end{itemize}
|
||||
\end{itemize}
|
||||
\textit{Scientific Workflow Management Systems} (SWMSs) are an essential tool for automating, managing, and executing complex scientific processes involving large volumes of data and computational tasks\footnote{citation?}. Traditional SWMSs employ a linear sequential approach, in which tasks are performed in a pre-defined order, as defined by the workflow. While this linear method is suitable for certain applications, it might not always be the best choice: processing sequentially can prove inefficient in cases where some processes can be done in parallel, or where the next step should adapt to the previous one. For these use-cases a dynamic scheduler is required, of which \textit{Managing Event Oriented Workflows}\autocite{DavidMEOW} (MEOW) is one.
|
||||
|
||||
MEOW employs an event-based scheduler, in which jobs are performed non-linearly, triggered based on events\footnote{citation?}. By dynamically adapting the execution order based on the outcomes of previous tasks or external factors, MEOW provides a more efficient and flexible solution for processing large volumes of experimental data\footnote{citation?}.
|
||||
|
||||
|
||||
\begin{tcolorbox}[colback=lightgray!30!white]
|
||||
What else should be included in the introduction?
|
||||
\end{tcolorbox}
|
||||
|
||||
\subsection{Problem}
|
||||
\textit{Something about the MEOW monitor, and expanding it. Probably also why it would be a good idea(?).}
|
||||
|
||||
\begin{itemize}
|
||||
\item Why \texttt{MEOW} needs it \begin{itemize}
|
||||
\item Further expansion of possibilities with the system
|
||||
\end{itemize}
|
||||
\end{itemize}
|
||||
In its current implementation, MEOW is able to trigger jobs based on changes to monitored files. This covers a wide range of scenarios, specifically where the data processing workflow involves the creation, modification, or removal of files. By monitoring file events, MEOW's event-based scheduler can dynamically execute tasks as soon as the required conditions are met, ensuring efficient and timely processing of the data.
|
||||
|
||||
While file events work well as a trigger on their own, there are several scenarios where a different trigger would be preferred or even required, especially when dealing with distributed systems or remote operations. To address these shortcomings and further enhance MEOW's capabilities, the integration of network event triggers would provide significant benefits in several key use-cases.
|
||||
|
||||
Firstly, network event triggers would allow for manual triggering of jobs without the need for direct access to the monitored files. This is particularly useful in scenarios where human intervention or decision-making is required before proceeding with the subsequent steps in a workflow. While it is possible to manually trigger job using file events by making changes to the monitored directories, this might lead to an already running job accessing the files at the same time, which could cause problems with data integrity.
|
||||
|
||||
Secondly, incorporating network event triggers would facilitate seamless communication between parallel jobs, ensuring that tasks can efficiently exchange information and synchronize their progress.
|
||||
|
||||
Finally, extending MEOW's event-based scheduler to support network event triggers would enable the simple and efficient exchange of data between workflows running on different machines. This feature is particularly valuable in distributed computing environments, where data processing tasks are often split across multiple systems to maximize resource utilization and minimize latency. By leveraging network event triggers, MEOW would be better equipped to manage complex workflows in these environments, ensuring seamless integration and streamlined data processing
|
||||
|
||||
\subsection{Background}
|
||||
\textit{Explaining the existing code and what it does and why. Also modules used and other important stuff to know.}
|
||||
\subsubsection{The structure of MEOW}
|
||||
The MEOW event-based scheduler has three main parts: \textit{monitors}, \textit{handlers}, and \textit{the conductor}.
|
||||
|
||||
\begin{itemize}
|
||||
\item General \texttt{MEOW\_BASE} structure: \begin{itemize}
|
||||
\item Monitors check if patterns are triggered \begin{itemize}
|
||||
\item Patterns are triggered by an event. Currently only file event triggers are implemented.
|
||||
\end{itemize}
|
||||
\item Handlers perform recipes \begin{itemize}
|
||||
\item Recipes are actions that are taken when triggered. python code can be run as a recipe.
|
||||
\end{itemize}
|
||||
\item The conductor handles the job queue of rules \begin{itemize}
|
||||
\item Rules combine patterns and recipes (When this pattern is triggered, perform this recipe).
|
||||
\end{itemize}
|
||||
\end{itemize}
|
||||
\item Specific (but not too granular) implementation details of \texttt{MEOW\_BASE}. \begin{itemize}
|
||||
\item Class structure
|
||||
\end{itemize}
|
||||
\item Other important stuff to know
|
||||
\end{itemize}
|
||||
Monitors monitor for triggering events. They are initialized with a number of \textit{patterns}, which describe the triggering event. When a pattern's triggering event occurs, the monitor signals to the conductor that the pattern has been triggered.
|
||||
|
||||
Handlers perform actions and jobs on behalf of the scheduler. They are initialized with a number of \textit{recipes}, which describe the action to be taken. The handler starts a job when signal to do so by the conductor.
|
||||
|
||||
The conductor handles the jobs queue. It is initialized with a number of rules, which a pattern paired with a recipe. When a monitor sends it a triggered pattern, the rules are checked for that pattern. If one or more rules contain that pattern, the corresponding recipes are triggered in their handler.
|
||||
|
||||
\subsubsection{The \texttt{meow\_base} codebase}
|
||||
\begin{tcolorbox}[colback=lightgray!30!white]
|
||||
Specific (but not too granular) implementation details of \texttt{meow\_base}.
|
||||
\end{tcolorbox}
|
||||
|
||||
\section{Method}
|
||||
\textit{Explaining the code I wrote and why I made those choices.}
|
||||
\begin{tcolorbox}[colback=lightgray!30!white]
|
||||
Explain the code I wrote and why I made those choices.
|
||||
|
||||
\begin{itemize}
|
||||
\item The \texttt{socket} library
|
||||
\item Expanding on existing code, reusing boiler-plate code
|
||||
\item Test-driven development
|
||||
\item Experiments with triggering on packet \begin{itemize}
|
||||
\item Removes the ability to send arbitrary data
|
||||
\end{itemize}
|
||||
\end{itemize}
|
||||
\end{tcolorbox}
|
||||
|
||||
\section{Results}
|
||||
\textit{Does it work? How well?}
|
||||
\begin{tcolorbox}[colback=lightgray!30!white]
|
||||
Does it work? How well?
|
||||
\end{tcolorbox}
|
||||
|
||||
\subsection{Testing}
|
||||
% \subsection{Testing}
|
||||
|
||||
\subsection{Discussion}
|
||||
\textit{With the hindsight of the results, what could I have done better?}
|
||||
\begin{tcolorbox}[colback=lightgray!30!white]
|
||||
With the hindsight of the results, what could I have done better?
|
||||
\end{tcolorbox}
|
||||
|
||||
\section{Future Work}
|
||||
\textit{What should someone do if they want to fix my mistakes, or expand on them further.}
|
||||
\begin{tcolorbox}[colback=lightgray!30!white]
|
||||
What should someone do if they want to fix my mistakes, or expand on them further.
|
||||
\end{tcolorbox}
|
||||
|
||||
\section{Conclusion}
|
||||
\textit{Did I succeed in what I wanted to do?}
|
||||
\begin{tcolorbox}[colback=lightgray!30!white]
|
||||
Did I succeed in what I wanted to do?
|
||||
\end{tcolorbox}
|
||||
|
||||
\newpage
|
||||
\appendix
|
||||
|
Reference in New Issue
Block a user