✨
This commit is contained in:
Binary file not shown.
@ -7,9 +7,11 @@
|
||||
% \usepackage[citestyle=verbose-ibid, backend=biber, autocite=footnote]{biblatex} % Footnote references. Use autocite{}.
|
||||
\usepackage{biblatex}
|
||||
\usepackage{float}
|
||||
\usepackage{fontspec}
|
||||
|
||||
% --- Configuration ---
|
||||
\bibliography{src/references}
|
||||
\setmonofont[Scale=0.85, ItalicFont=Hermit Light]{Hermit Light}
|
||||
% \pagestyle{fancy}
|
||||
% \setlength{\parskip}{6pt}
|
||||
% \setlength{\parindent}{0pt}
|
||||
@ -65,7 +67,7 @@
|
||||
\begin{center}
|
||||
\includegraphics[width=\textwidth]{src/heterogeneous.png}
|
||||
\end{center}
|
||||
\caption{\textbf{Temp.} An example of a heterogeneous workflow}
|
||||
\caption{An example of a heterogeneous workflow}
|
||||
\end{figure}
|
||||
|
||||
\subsection{Background}
|
||||
@ -82,7 +84,7 @@
|
||||
\caption{The monitor's role in MEOW's event-based system.}
|
||||
\end{figure}
|
||||
|
||||
\begin{tcolorbox}[colback=lightgray!30!white]
|
||||
\begin{tcolorbox}[colback=blue!30!white]
|
||||
I haven't used "Resources" to describe the job queue. Should I do that or should I rephrase the diagram to be more in line with the rest of the project?
|
||||
\end{tcolorbox}
|
||||
|
||||
@ -113,40 +115,41 @@
|
||||
|
||||
\subsubsection{The \texttt{socket} library}
|
||||
|
||||
The socket library\autocite{SocketDoc}, included in the Python Standard Library, serves as an interface for the Berkeley sockets API. The Berkeley sockets API, originally developed for the Unix operating system, has become the standard for network communication across multiple platforms. It allows programs to create 'sockets', which are endpoints in a network communication path, for the purpose of sending and receiving data.
|
||||
The \texttt{socket} library\autocite{SocketDoc}, included in the Python Standard Library, serves as an interface for the Berkeley sockets API. The Berkeley sockets API, originally developed for the Unix operating system, has become the standard for network communication across multiple platforms. It allows programs to create 'sockets', which are endpoints in a network communication path, for the purpose of sending and receiving data.
|
||||
|
||||
\begin{tcolorbox}[colback=lightgray!30!white]
|
||||
\begin{itemize}
|
||||
\item What am I using it for?
|
||||
\item Which parts am I using?
|
||||
\item Other options? \begin{itemize}
|
||||
\item \texttt{ssl}
|
||||
\item various API and web frameworks
|
||||
\end{itemize}
|
||||
\end{itemize}
|
||||
Many other libraries and modules focusing on transferring data exist for Python, some of which may be better in certain MEOW use-cases. The \texttt{ssl} library, in specific, allows for ssl-encrypted communication, which may be a requirement in workflows with sensitive data. However, implementing network triggers using the \texttt{socket} library will provide MEOW with a basic implementation of network events, which can later be expanded or improved with other features.
|
||||
|
||||
In my project, all sockets use the Transmission Control Protocol (TCP), which ensures safe data transfer by enforcing a stable connection between the sender and receiver. I make use of the following socket methods, which have the same names and functions in the \texttt{socket} library and the Berkeley sockets API:
|
||||
|
||||
\begin{tcolorbox}[colback=blue!30!white]
|
||||
Too granular?
|
||||
\end{tcolorbox}
|
||||
|
||||
\begin{itemize}
|
||||
\setlength{\itemsep}{-5pt}
|
||||
\item \texttt{bind()}: Associates the socket with a given local IP address and port. It also reserves the port locally.
|
||||
\item \texttt{listen()}: Puts the socket in a listening state, where it waits for a sender to request a TCP connection to the socket.
|
||||
\item \texttt{accept()}: Accepts the incoming TCP connection request, creating a connection.
|
||||
\item \texttt{recv()}: Receives data from the given socket.
|
||||
\item \texttt{connect()}: Sends a TCP connection request to a listening socket. This is only used in testing the monitor.
|
||||
\item \texttt{sendall()}: Sends data a socket. This is only used in testing the monitor.
|
||||
\item \texttt{close()}: Closes a connection to a given socket.
|
||||
\end{itemize}
|
||||
|
||||
\section{Method}
|
||||
|
||||
To address the identified limitations of MEOW and to expand its capabilities, I will be incorporating network event triggers into the existing event-based scheduler, to supplement the current file-based event triggers. My method focuses on leveraging Python's socket library to enable the processing of network events. The following subsections detail the specific methodologies employed in expanding the codebase, the design of the network event trigger mechanism, and the integration of this mechanism into the existing MEOW system.
|
||||
|
||||
\subsection{Design of the network event pattern}
|
||||
\begin{tcolorbox}[colback=lightgray!30!white]
|
||||
\begin{itemize}
|
||||
\item Expanding on existing code, reusing boiler-plate code
|
||||
\item Attempts to preserve loose coupling of modules (any trigger should be able to connect to any handler) (this might not be entirely possible, but it's a good idea to attempt)
|
||||
\item Experiments with triggering on packet \begin{itemize}
|
||||
\item Removes the ability to send arbitrary data
|
||||
\end{itemize}
|
||||
\item To simplify, raw data is expected
|
||||
\end{itemize}
|
||||
\end{tcolorbox}
|
||||
A main concern with implementing a pattern for network events is to seamlessly integrate it with the existing codebase. Because of this, the design of the pattern has a heavy focus on behaving similarly to the file event pattern when interacting with the other elements of the scheduler. Ideally, this should preserve loose coupling of the patterns and recipes, so any pattern can be put in a rule with any recipe. While this might not be possible for every theoretical recipe and pattern, designing for it could greatly improve future compatibility.
|
||||
|
||||
Network event patterns are initialized with a triggering port, similar to the triggering path of the file event patterns. While this limits the amount of possible unique patterns to the amount of ports that can be opened on the machine, that amount is large enough that it will likely not be an issue. It would have been possible to have the patterns be triggered by part of the sent message, acting as a "header". However, this would complicate the process, since the monitor will otherwise be expecting to receive raw data. This was chosen in order for the implementation to be as simple as possible, so that any feature or improvement can be added later as its own pattern type.
|
||||
|
||||
The network monitor, when started, opens sockets that start listening on the ports specified in the patterns it was initialized with.
|
||||
|
||||
\subsection{Integrating it into the existing codebase}
|
||||
Data received by the network monitor is written to a temporary file, which serves two purposes. Firstly, writing the received data to a file while receiving it saves on memory, since the entire file doesn't have to be saved in memory at once. This is especially useful for large data transfers. Secondly, writing the received data to a file allows network events to reuse most of the infrastructure written for file events, passing the newly written temporary file as the "triggering path" of the event. This means that recipes taking the triggering path as their input can still be used with network events, preserving loose coupling.
|
||||
|
||||
\begin{tcolorbox}[colback=lightgray!30!white]
|
||||
Reusing the system for file event triggering by way of temp files.
|
||||
\end{tcolorbox}
|
||||
\subsection{Testing}
|
||||
|
||||
\section{Results}
|
||||
@ -166,6 +169,7 @@
|
||||
What should someone do if they want to fix my mistakes, or expand on them further.
|
||||
\begin{itemize}
|
||||
\item Implementation of the other options mentioned when discussing the socket library.
|
||||
\item Triggering on a header item in addition to port
|
||||
\end{itemize}
|
||||
\end{tcolorbox}
|
||||
|
||||
@ -178,7 +182,7 @@
|
||||
\begin{center}
|
||||
\includegraphics[width=0.6\textwidth]{src/BIDS.png}
|
||||
\end{center}
|
||||
\caption{\textbf{Temp graph. Replace.} The structure of the BIDS workflow. Data is transferred to user, and to the cloud.}
|
||||
\caption{\textbf{Temp.} The structure of the BIDS workflow. Data is transferred to user, and to the cloud.}
|
||||
\end{figure}
|
||||
|
||||
\section{Conclusion}
|
||||
|
Reference in New Issue
Block a user