In the realm of aerospace engineering, ensuring the reliability and longevity of critical components is paramount. The ability to predict and prevent potential failures in aircraft engines can not only enhance safety but also significantly reduce operational costs. To address these challenges, researchers and engineers have turned to advanced technologies such as machine learning to develop predictive maintenance solutions.

Introduction to Predictive Maintenance

Predictive maintenance is a proactive approach to maintenance management that leverages data analytics and machine learning algorithms to predict equipment failures before they occur. By analyzing historical data and real-time sensor readings, predictive maintenance models can identify patterns and anomalies indicative of impending failures, allowing maintenance activities to be scheduled precisely when needed.

• Introduction

• Data Collection and Preparation

  Importing Libraries and Loading Data

  Data Preprocessing

  Data Augmentation

• Building the CNN Model

  Model Architecture

  Compiling the Model

  Training the Model

• Model Evaluation

• Performance Metrics

  Visualizing Training History

  Model Deployment

• Conclusion

INTRODUCTION:

Agricultural 🌱 practices play a crucial role in ensuring global food security. However, the health of crops can be significantly impacted by various factors, including the presence of diseases. One such crop of immense importance is the potato, a staple food for many populations around the world. Potatoes are susceptible to diseases such as Early Blight, Late Blight, and can also exhibit a healthy state.

In this context, leveraging advanced technologies becomes imperative to efficiently monitor and manage the health of potato crops. Deep learning, particularly Convolutional Neural Networks (CNNs), has shown remarkable success in image recognition tasks. This project focuses on utilizing CNNs for the detection and classification of diseases in potato leaves, specifically targeting Early Blight, Late Blight, and Healthy states.

We will build a web application to predict the diseases of Potato plants.

This application will help farmers to identify the diseases in potato plants so that they can use appropriate fertilizers to get more yield.We will build a web application to predict the diseases of Potato plants.

This application will help farmers to identify the diseases in potato plants so that they can use appropriate fertilizers to get more yield.

Image Classification

A classical computer vision problem, where the task is to predict the class of an image within a known set of possible classes.

Problem statement

• To classify the given potato leaf image as healthy, late blight or early blight.

• It is a multi class classification problem.

Data

We will use a kaggle dataset for this project.
I created a subset of the original data, which includes only the diseases of potato plants. You can find the dataset used in this project

Late Blight: Late blight of potato is a disease caused by fungus Phytophthora infestans.

Early Blight: Early blight of potato is a disease caused by the fungus Alternaria solani

Healthy: Uninfected or healthy plant

Data Collection and Preparation

Importing Libraries and Loading Data

involves importing the necessary libraries for the project, such as TensorFlow and Matplotlib, and loading the dataset of potato images, it ensures that all required dependencies are available and accessible for further processing.

import tensorflow as tf
from tensorflow.keras import models, layers
import matplotlib.pyplot as plt

This part imports the necessary libraries, including TensorFlow and Matplotlib, and loads the datasets of potato images using TensorFlow's image_dataset_from_directory function.

Data Preprocessing

Data preprocessing involves preparing the dataset for model training by performing operations such as resizing, normalization, and batching. This section ensures that the data is in the appropriate format and structure for training the CNN model.

IMAGE_SIZE = 256
BATCH_SIZE = 32
CHANNELS = 3
EPOCHS = 50

Defines constants for image size, batch size, number of color channels, and number of epochs.

Data Augmentation

Data augmentation is a technique used to artificially increase the diversity of the training dataset by applying transformations such as rotation, flipping, and scaling to the images. This helps the model generalize better and improves its performance on unseen data.

data_augmentation = tf.keras.Sequential([
    layers.experimental.preprocessing.RandomFlip("horizontal_and_vertical"),
    layers.experimental.preprocessing.RandomRotation(0.2)
])

Building the CNN Model

Model Architecture

The model architecture defines the structure and configuration of the CNN model, including the number and type of layers, their activation functions, and their connectivity. It lays the foundation for the neural network's ability to learn and make predictions based on input data.

input_shape = (BATCH_SIZE, IMAGE_SIZE, IMAGE_SIZE, CHANNELS)
n_classes = 3

model = models.Sequential([
    resize_and_rescale,
    data_augmentation,
    layers.Conv2D(32, kernel_size=(3,3), activation='relu', input_shape=input_shape),
    layers.MaxPooling2D((2, 2)),
    layers.Conv2D(64,  kernel_size=(3,3), activation='relu'),
    layers.MaxPooling2D((2, 2)),
    layers.Conv2D(64,  kernel_size=(3,3), activation='relu'),
    layers.MaxPooling2D((2, 2)),
    layers.Conv2D(64, (3, 3), activation='relu'),
    layers.MaxPooling2D((2, 2)),
    layers.Conv2D(64, (3, 3), activation='relu'),
    layers.MaxPooling2D((2, 2)),
    layers.Conv2D(64, (3, 3), activation='relu'),
    layers.MaxPooling2D((2, 2)),
    layers.Flatten(),
    layers.Dense(64, activation='relu'),
    layers.Dense(n_classes, activation='softmax'),
])

model.build(input_shape=input_shape)
model.summary()

Defines the architecture of the CNN model using TensorFlow's Sequential API.

The next step is to investigate model architecture.

Let’s have a look at the brief summary of our mode

Compiling the Model

Compiling the model involves configuring its learning process by specifying the optimizer, loss function, and evaluation metrics. This step prepares the model for training by defining how it should update its parameters to minimize the loss and improve performance.

model.compile(
    optimizer='adam',
    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),
    metrics=['accuracy']
)

Compiles the model with the Adam optimizer, sparse categorical crossentropy loss function, and accuracy metric.

Training the Model

Training the model involves feeding the training dataset into the neural network and iteratively adjusting its parameters to minimize the loss function. This process allows the model to learn from the data and improve its ability to make accurate predictions.

history = model.fit(
    train_ds,
    batch_size=BATCH_SIZE,
    validation_data=val_ds,
    verbose=1,
    epochs=EPOCHS,
)

Trains the model on the training dataset and evaluates its performance on the validation dataset.

Model Evaluation

Model evaluation involves assessing the performance of the trained model on separate validation and test datasets. It provides insights into how well the model generalizes to unseen data and helps identify areas for improvement

scores = model.evaluate(test_ds)

Evaluates the model on the test dataset and prints the evaluation scores.

Performance Metrics

Performance metrics are quantitative measures used to evaluate the effectiveness of the trained model. Common metrics include accuracy, precision, recall, and F1 score, which provide information about the model's classification performance.

history.history.keys()

Prints the keys available in the history object, which contains training and validation metrics.

Visualizing Training History

Visualizing the training history involves plotting graphs of training and validation metrics, such as accuracy and loss, over the course of training epochs. This visualization helps identify trends, patterns, and potential issues in the model's learning process.

plt.plot(range(EPOCHS), acc, label='Training Accuracy')
plt.plot(range(EPOCHS), val_acc, label='Validation Accuracy')
plt.title('Training and Validation Accuracy')
plt.legend(loc='lower right')
plt.show()

Plots the training and validation accuracy over epochs to visualize the model's learning progress.

Let’s have a look at the history parameter.

Actually, history is a Keras callback that keeps all epoch history as a list; let’s utilize it to plot some intriguing plots. Let’s start by putting all of these parameters into variables.

This graph shows the accuracy of training vs validation. Epochs are on the x-axis, and accuracy and loss are on the y-axis.

Let's save our model 
# it will save the model
model.save('final_model.h5')

Model Deployment

Saving the Model

Saving the model involves exporting its architecture and trained weights to a file for future use or deployment. This allows the model to be easily loaded and used in other applications without needing to retrain it from scratch.

model.save("potato_model.h5")

Saves the trained model to a file for future use or deployment.

Streamlit – The Boom!

Streamlit is a free, open-source Python Framework, that allows us to quickly develop a Web Application without the requirement of a backend server and without having to write HTML, CSS, or Javascript. We can start building a really good Web Application simply by using our existing python skills. I’ve created a simple web application that accepts images as input and requires the same preprocessing steps on the input image as we did on our training dataset during training because when we save our model, it only saves model trained parameters, and we must preprocess our input manually, so this is something we must keep in mind when building any web application or using a pre-trained model.

Web App

import streamlit as st
from PIL import Image
import numpy as np
import tensorflow as tf
from werkzeug.utils import secure_filename
import os

os.environ["CUDA_VISIBLE_DEVICES"] = "-1"

class_names = ['Potato___Early_blight', 'Potato___Late_blight', 'Potato___healthy']

def predict(model, img):
    img_array = tf.keras.preprocessing.image.img_to_array(img)
    img_array = tf.expand_dims(img_array, 0)
    predictions = model.predict(img_array)
    predictions_arr = [round(100 * i, 2) for i in predictions[0]]
    predicted_class = class_names[np.argmax(predictions[0])]
    confidence = round(100 * (np.max(predictions[0])), 2)
    return predicted_class, predictions_arr

model = tf.keras.models.load_model('potato_model.h5', compile=False)

def main():
    st.set_page_config(page_title="Potato Disease Classifier")
    st.sidebar.title("Potato Disease Classifier")
    st.sidebar.info("Upload an image of a potato leaf to detect early or late blight.")
    st.title("Potato Disease Detection")
    uploaded_file = st.file_uploader("Upload a potato leaf image",type=['jpg','png','jpeg'])
    if uploaded_file is not None:
        image = Image.open(uploaded_file)
        st.image(image,caption="Uploaded Image",use_column_width=True)
        image = image.resize((256,256))
        img_arr = np.array(image)
        predicted_class,predictions=predict(model,img_arr)

        response = {
            "predicted_class": predicted_class,
            "early": f"{predictions[0]:.2f}%",
            "late": f"{predictions[1]:.2f}%",
            "healthy": f"{predictions[2]:.2f}%"
        }


        st.success(f"Predicted Class : {response['predicted_class']}",icon="✅")
        st.write("Probabilities:")
        col1,col2,col3 = st.columns(3)
        col1.metric("Early Blight" , f"{response['early']}", f"{response['early']}")
        col2.metric("Late Blight" , f"{response['late']}", f"{response['late']}")
        col3.metric("Healthy" , f"{response['healthy']}", f"{response['healthy']}")

if __name__ == "__main__":
    main()

Output :

Internally, the web app uses our previously developed deep learning model to detect potato leaf diseases.

Potato early bright

Potato healthy

Potato late blight

Conclusion

As we conclude this chapter of the Moroccan Agriculture Management System project, we're reminded of the impactful results that collaborative efforts, like those facilitated by the MDS community, can achieve. By harnessing data and technology, we gain insights into societal trends and cultural phenomena, enabling informed decision-making, community engagement, and positive social change.

Moving forward, let's continue embodying the spirit of collaboration, curiosity, and inclusivity that defines the MDS community. Together, we'll persist in exploring, innovating, and inspiring, shaping a brighter future through data science and community-driven initiatives.

Acknowledgments

I Want to express my heartfelt appreciation to my dedicated team members Nizar Sahl, Idriss EL HOUARI, Farheen Akhter, Ben alla ismail, and Aicha Dessa. Their expertise and commitment have been invaluable to this project. Your hard work, collaboration, and enthusiasm have truly made a difference. As the team leader, I am incredibly proud to have worked alongside such talented individuals.

I also want to acknowledge Bahae Eddine HALIM, the founder of the Moroccan Data Science MDS Community for providing the platform for our project through the "DataFtour" second Edition initiative. His dedication to fostering a supportive environment for data enthusiasts in Morocco has been instrumental in our journey. Lastly, we thank the broader data science community for their support and encouragement, which have motivated us to push boundaries and continuously strive for excellence.

Explore a preview of our project:

HuggingFace

Github

You may connect with me Linkedin also follow me.

Thank you ✨🧠🌱

MDS Community

In the vibrant landscape of data science and community-driven initiatives, the MDS community stands as a beacon of innovation and collaboration. Founded by Bahae Eddine Halim , the MDS community has brought together a diverse group of passionate individuals, united by their shared commitment to leveraging data science for societal impact.

In this article, we delve into one of the latest endeavors spearheaded by the MDS community, a groundbreaking project aimed at analyzing Moroccan online discourse during Ramadan. With the collective efforts of Bahae Eddine Halim ,Loubna Bouljadiane , Soufiane sejjari ,Zineb El houz ,Ali ,Hiba Lbazry, and Imad Nasri , this project embarks on a journey to uncover insights from the vast sea of Moroccan Darija comments and tweets.

At its core, the main objective of this project is to harness the power of data science, machine learning, and deep learning to decode the sentiments, topics, and engagement patterns prevalent in Moroccan online conversations during the sacred month of Ramadan. Through meticulous analysis and cutting-edge methodologies, we aim to shed light on the nuanced perspectives and cultural dynamics shaping Moroccan society in the digital age.

Explore a preview of our project : https://moroccansentimentsanalysis.com

Explore the world of Ramadan comments in Moroccan Darija through EDA, uncovering patterns, trends, and insights from social media platforms like Facebook, Twitter, Hespress, and YouTube. Discover the prevalence of religious themes, temporal engagement patterns, language distribution, topic modeling results, and comment length analysis across different platforms. Gain valuable insights into online discourse during Ramadan and the significance of EDA in extracting meaningful insights from diverse datasets.

Data Scraping

As we know in the world of AI the most important thing is the data , so in fact having lots of data is like having a big toolbox - the more tools you have , the easier it is to fix things.

So before that we dive into how we scrap our data from different sources like YouTube, Twitter(X) ... let's understand scraping.

What is the scraping ?

" Scraping is like gathering ingredients for a recipe, but from websites instead of the grocery store " .

Scraping is simply the process of extracting data from websites. It allows us to collect information for various purposes such as research or building datasets for analysis. In the context of our project, we're specifically interested in scraping comments written in Darija (Moroccan language) from different sources.

How do we get data from the websites ?

There are several methods to perform web scraping, each with its own advantages and limitations. Here are some common methods along with brief explanations:

1. Manual Scraping: This involves manually copying and pasting data from websites into a local file or database. While simple and straightforward, it is not practical for scraping large amounts of data and is highly inefficient.

2. Beautiful Soup: Beautiful Soup is a Python library designed for quick and easy scraping of web pages. It provides a simple API for navigating and searching the HTML structure of a webpage, making it ideal for extracting specific information from websites.

3. Selenium: Selenium is another popular tool for web scraping, particularly useful for scraping dynamic content generated by JavaScript. It allows automation of web browsers to interact with web pages, enabling scraping of content that is rendered dynamically.

4. API Scraping: Many websites offer APIs (Application Programming Interfaces) that allow developers to access their data in a structured and legal manner. By interacting with these APIs, developers can retrieve data without having to parse HTML or deal with web scraping complexities.

Now, let's relate these methods to our project:

From Hespress :

• We employed a combination of Beautiful Soup and Selenium techniques to extract data from the HESPRESS website. The data we scraped includes the article's title, published date, and comments associated with the article.

the process :

1. Importing Libraries:

   • The webdriver module from Selenium serves a pivotal role in automating interactions with web browsers, enabling developers to programmatically navigate and interact with web pages. This powerful tool empowers users to simulate user actions such as clicking buttons, filling out forms...

     On the other hand, BeautifulSoup, imported from the bs4 library, provides a versatile and user-friendly solution for parsing and extracting data from HTML and XML files.

         import os
         import time
         from selenium import webdriver
         from bs4 import BeautifulSoup
     

2. Web Scraping Process:

   • The process begins by creating a webdriver.Chrome instance, which specifies the path to the Chrome WebDriver executable (chromedriver). This instance acts as a bridge between the Python script and the Chrome browser, enabling automated interactions with web pages.

     Next, the browser is directed to the specified URL using the driver.get(url) command.

     To ensure that the page is fully loaded before proceeding with further actions, a delay of 3 seconds is introduced using time.sleep(3).

     Once the page is fully loaded, the HTML source code of the webpage is retrieved using driver.page_source.

     Finally, the retrieved HTML source code is passed to BeautifulSoup for parsing and analysis. Using the BeautifulSoup(src, 'lxml') syntax,

         driver = webdriver.Chrome('C:/Users/sejja/chromedriver')  # Assuming valid path
         driver.get(url)
         time.sleep(3)
     
         src = driver.page_source
         soup = BeautifulSoup(src, 'lxml')
     

   • Various elements of interest, such as the title, date, tags, and comments, are extracted from the parsed HTML using BeautifulSoup's find and findAll methods. Here is an example how we extract title and the comments :

         titre = soup.find('h1', {'class': 'post-title'})
                 if titre:
                     titre = titre.get_text().strip()
                 else:
                     titre = "not available"
     
                 comments_area = soup.find('ul', {'class': 'comment-list hide-comments'})
                 comments = []
                 if comments_area:
                     for comment in comments_area.findAll('li', {'class': 'comment even thread-even depth-1 not-reply'}):
                         comment_date = comment.find('div', {'class': 'comment-date'})
                         comment_content = comment.find('div', {'class': 'comment-text'})
                         comment_react = comment.find('span', {'class': 'comment-recat-number'})
                         if comment_date and comment_content and comment_react:
                             comments.append({
                                 "comment_date": comment_date.get_text(),
                                 "comment_content": comment_content.get_text(),
                                 "comment_react": comment_react.get_text()
                             })
     
                 return {'Date': date, 'Titre': titre, 'Tags': tags, 'Comments': comments}
     

And finally extracted data is stored in a dictionary format.

    return {'Date': date, 'Titre': titre, 'Tags': tags, 'Comments': comments}

3. Cleanup:

   • driver.quit() ensures that the WebDriver browser instance is closed, even if an exception occurs, preventing resource leaks.

From YouTube :

YouTube API: The YouTube API provides developers with programmatic access to YouTube's features, including retrieving video metadata, comments, and other relevant data.

The code first fetches details about the specified video, including the channel name and video title. Then, it retrieves comments associated with the video using the 'commentThreads' endpoint. To handle pagination, it iterates through multiple pages of comments, ensuring that all comments are captured.

The extracted data includes essential information such as the video title, channel name, comment date, content, likes, dislikes, author, and number of replies.

the process :

• Get the YouTube API :

  1. To obtain your YouTube Data API key, you need to follow these steps:

     1. Sign in to Google:

     - Go to the Google Developers Console at console.developers.google.com

     - Sign in with your Google account. If you don't have one, you'll need to create it.

     2. Create a new project:

     - If you don't have any existing projects, you'll be prompted to create one. Click on the "Select a project" dropdown menu at the top and then click on the "New Project" button.

     - Enter a name for your project and click on the "Create" button.

     3. Enable the YouTube Data API:

     - In the Google Cloud Console, navigate to the "APIs & Services" > "Library" page using the menu on the left.

     - Search for "YouTube Data API" in the search bar.

     - Click on the "YouTube Data API v3" result.

     - Click on the "Enable" button.

     4. Create credentials:

     - After enabling the API, navigate to the "APIs & Services" > "Credentials" page using the menu on the left.

     - Click on the "Create credentials" button and select "API key" from the dropdown menu.

     - Your API key will be created. Copy it and securely store it.

     5. Use your API key:

     - Now that you have your API key, you can use it in your applications to access the YouTube Data API.

from googleapiclient.discovery import build
import pandas as pd

# YouTube Data API key
API_KEY = 'your YouTube API key'

• build the YouTube service object :

 # Build the YouTube service object . It requires specifying the API name ('youtube'), API version ('v3'), and developer API key (API_KEY) obtained from the Google Developer Console.
youtube_service = build('youtube', 'v3', developerKey=API_KEY)

• retrieve video details

 # Retrieve video details using the 'videos' endpoint
video_response = youtube_service.videos().list(
       part='snippet',
       id=video_id
).execute()

• Extract the essential information such as the video title, channel name, comments :

here we extracted channel name and video title from the video_response :

# Extract channel name and video title
channel_name = video_response['items'][0]['snippet']['channelTitle']
video_title = video_response['items'][0]['snippet']['title']

and here we got the comments plus some details such as the comment's date, numbers of likes , the author ... : (including the pagination)

response = youtube_service.commentThreads().list(
          part='snippet,replies',
          videoId=video_id,
          textFormat='plainText',
          maxResults=800,  # Increase the max results per page if necessary
          pageToken=nextPageToken
     ).execute()

if not response['items']:
        print(f"Comments are disabled for video: {video_id}")
        return []

for item in response['items']:
        snippet = item['snippet']['topLevelComment']['snippet']
        comments.append({
                'Video Title': video_title,
                'Channel Name': channel_name,
                'Comment Date': snippet.get('publishedAt', ''),
                'Comment': snippet.get('textDisplay', ''),
                'Likes': snippet.get('likeCount', 0),
                'Dislikes': snippet.get('dislikeCount', 0),
                'Author': snippet.get('authorDisplayName', ''),
                'Replies': item['snippet']['totalReplyCount']
         })

• Search for YouTube videos based on a query string:

So we added a function to allow us to search for YouTube videos based on a query string, with optional filters for region (for us we used MA indicating the Moroccan region) and publication dates (matching with Ramadan month). It fetches video IDs matching the search criteria, which can be further used to retrieve additional details or perform other operations on the videos.

For Twitter :

• Twscrape : It is a tool for scraping data from tweets. It collects data such as user profiles, follower lists and follower lists, likes and retweets, as well as keyword searches.

The extracted data includes essential information such as the username, content and comment date.

The process :

• Fetching Tweets: Utilizing an asynchronous function gather_tweets() to retrieve tweets from an asynchronous Twitter scraper. It's similar to YouTube , we used a query string to retrieve the tweets related to Ramadan month. And we need also information about twitter (X) account (username, password ..)

      import asyncio
      import twscrape
      from twscrape import API, gather
      import pandas as pd
  
      class TwitterScraper:
          def __init__(self):
              self.api = API()
  
          async def gather_tweets(self, query="رمضان", limit=20):
              await self.api.pool.add_account("username dial twitter", "password", "email","mail_pass") #mail pass optionel  , we can have more than 1
              await self.api.pool.login_all()
  
              tweets = await gather(self.api.search(query, limit=limit))
  
              data = []
              for tweet in tweets:
                  tweet_data = {
                      'ID': tweet.id,
                      'Username': tweet.user.username,
                      'Content': tweet.rawContent,
                      'Date': tweet.date
                  }
                  data.append(tweet_data)
  
                  print(tweet.id, tweet.user.username, tweet.rawContent)
  
              df = pd.DataFrame(data)
              return df
  

• Displaying Tweets: Tweets are displayed on the screen during the retrieval process.

• Saving Tweets: Tweet data is saved in either a CSV or JSON file, depending on your choice.

For Facebook :

• As we marked before for Hespress website, we also used for Facebook a combination of Beautiful Soup and Selenium techniques to extract data from it. The data we scraped includes the article's title, published date, and comments associated with the article.

the process :

1. Import necessary libraries:

      from bs4 import BeautifulSoup
      from selenium import webdriver
      from selenium.common.exceptions import NoSuchElementException, TimeoutException
      from selenium.webdriver.common.by import By
      import time
      import pandas as pd
   

2. Get the posts from a Facebook page :

   Define the function get_facebook_post_data to get posts from a Facebook page. Inside this function we initialize a Chrome WebDriver assuming the path to chromedriver is valid. Navigate to the provided Facebook page URL. Click the "See More Posts" button if present. Scroll down to load more posts based on the scroll_count. Use BeautifulSoup to parse the page source. Extract post data (title, link, date, reactions, comments) using specified class names. And finally construct a DataFrame from the extracted data and returns it.

   Usage example:

      series = get_facebook_post_data('https://web.facebook.com/alwa3d4', scroll_count=80)
   

3. Usefacebook_scraperlibrary to extract data from Facebook posts :

   1. Import required libraries:

         import facebook_scraper as fs
         import pandas as pd
         from facebook_scraper import exceptions  # Import specific exceptions
      

   2. Define the FacebookScraper class:

      • Constructor that Initializes the maximum number of comments to retrieve (MAX_COMMENTS).

      • We have also the Method getPostData that have the parameter post_url URL of the Facebook post. This function extracts the post ID from the provided URL, attempts to retrieve post data using facebook_scraper, handles potential errors such as missing comments or invalid URLs. And if comments are found, normalizes the JSON data into a DataFrame.

            post_id = post_url.split("/")[-1].split("?")[0]  # Extract post ID
                        print(post_id)
        
                        # Attempt to get post data, handling potential errors
                        gen = fs.get_posts(post_urls=[post_id], options={"comments": self.MAX_COMMENTS, "progress": True})
                        post = next(gen)
        
                        # Handle missing 'comments_full' key
                        comments = post.get('comments_full', [])  # Use default empty list if missing
        
                        if comments:
                            df = pd.json_normalize(comments, sep='_')
                            return df
                        else:
                            print(f"No comments found for post: {post_id}")
                            return None  # Return None to indicate no comments
        

In conclusion, data scraping is a crucial process in collecting data from various sources like websites and social media platforms. By utilizing tools like Beautiful Soup, Selenium, and APIs, developers can efficiently gather and analyze data for research and analysis purposes. The ability to extract valuable information from websites, YouTube, Twitter, and Facebook opens up opportunities for AI projects to utilize data effectively.

Data Cleaning

After successfully scraping data from various sources such as Hespress, YouTube, Twitter, and Facebook, the next crucial step is cleaning this diverse dataset. Each source brings its own set of challenges and characteristics, making data cleaning essential to ensure accuracy and reliability in subsequent analysis .In our cleaning process we used various techniques to get the data ready for analysis.

Hespress Dataset

Data structure : The Hespress dataset comprises columns including Date, Titre, Tags, and Comments. The Date column records the date and time of comment posting, presented in Arabic format (e.g., "الجمعة 29 مارس 2024 - 18:00"). Titre represents the title or headline of the corresponding content. Tags denote keywords or labels associated with the content. Comments, depicted as a string format dictionary, encapsulate comment-related data, including comment_date, comment_content, and comment_react. Each comment_date entry denotes the timestamp of comment posting, while comment_content encapsulates the textual content of the comment. Lastly, comment_react records reactions or feedback received on the comment

import pandas as pd 
hes_data1 = pd.read_csv("/content/drive/MyDrive/DataSets sentiment analysis/hespressComments.csv")
hes_data2 = pd.read_csv("/content/drive/MyDrive/DataSets sentiment analysis/hespressComments2.csv")

We focused solely on two variables: Date and Comments

hes_data2 = hes_data2[['Date','Comments']]
hes_data1 = hes_data1[['Date','Comments']]

Data cleaning :

the Comments variable, initially presented as a string format containing a list of dictionaries, encapsulates essential comment-related data, including comment_date, comment_content, and comment_react. To facilitate further analysis, a crucial step involved converting this string format into a list of dictionaries.

import ast
def convert_str_to_list_of_dicts(input_str):
    try:
        result_list = ast.literal_eval(input_str)
        if isinstance(result_list, list):    
            if all(isinstance(item, dict) for item in result_list):
                return result_list
    except (SyntaxError, ValueError):
        pass
    return []
hes_data1['Comments'] = hes_data1['Comments'].apply(convert_str_to_list_of_dicts)
hes_data2['Comments'] = hes_data2['Comments'].apply(convert_str_to_list_of_dicts)
hes_data = pd.concat([hes_data1,hes_data2])

In addition to converting the Comments variable from a string format containing a list of dictionaries to an actual list of dictionaries, another crucial step involved data validation. Some comments were found to contain empty lists, which could potentially skew the analysis. Therefore, I implemented a step to identify and remove these empty lists from the dataset. This data validation process ensured the integrity and reliability of the dataset for subsequent sentiment analysis

hes_data = hes_data[hes_data['Comments'].apply(lambda x: len(x) > 0)]
hes_data.reset_index(drop=True, inplace=True)

We transformed the Comments variable from a list of dictionaries into a structured Data Frame. This involved organizing each comment's date and content into a single Data Frame for streamlined analysis. By leveraging Python's Pandas library. This transformation facilitated easier access and interpretation of the comment data, laying the groundwork for subsequent sentiment analysis.

comments_list = []
for i, comment_data in enumerate(hes_data['Comments']):
    for comment_dict in comment_data:
        comment_date = comment_dict['comment_date'].strip()
        comment_content = comment_dict['comment_content'].strip()

        comments_list.append({'Date': comment_date, 'Comment': comment_content})


hes_data_final = pd.DataFrame(comments_list)

The next step in the data cleaning process involved transforming the date format from its original Arabic format (e.g., 'الجمعة 29 مارس 2024 - 18:00') to a standardized format ('YYYY-MM-DD HH:MM:SS'). This conversion ensured consistency and compatibility with common date-time formats, facilitating easier manipulation and analysis of the data. By implementing this transformation, the dataset's date information was brought into a uniform structure, ready for further analysis and visualization.

from datetime import datetime

def convert_date(date_str):    
    parts = date_str.split()
    day = int(parts[1])
    month = {
        'يناير': 1, 'فبراير': 2, 'مارس': 3, 'أبريل': 4, 'ماي': 5, 'يونيو': 6,
        'يوليوز': 7, 'غشت': 8, 'شتنبر': 9, 'أكتوبر': 10, 'نونبر': 11, 'دجنبر': 12
    }[parts[2]]
    year = int(parts[3])
    time = parts[-1]
    date_time = datetime(year, month, day)
    time_parts = time.split(':')
    date_time = date_time.replace(hour=int(time_parts[0]), minute=int(time_parts[1]))
    return date_time

hes_data_final['Date'] = comments_df['Date'].apply(convert_date)

In addition to the transformation of the date format, I added a new column named 'source' to the Data Frame and assigned it the value 'Hespress' for each row. This step allowed for easy identification and categorization of the data based on its source. By including this metadata, the dataset became more informative and well-organized, facilitating subsequent analysis and interpretation.

hes_data_final = hes_data_final.assign(source = 'Hespress')

To enhance the dataset's comprehensiveness, the final step involved detecting the language of each comment. Utilizing the 'langdetect' library, we automatically identified the language of the comment text. This process enabled us to distinguish between comments in different languages, ensuring that subsequent analysis accurately reflected the linguistic diversity of the dataset. By incorporating language detection, we enriched the dataset with valuable metadata, facilitating more nuanced insights and interpretation

pip install langdetect

from langdetect import detect, DetectorFactory
DetectorFactory.seed = 0
def detect_language(comment):
    try:
        lang = detect(comment)
        return lang
    except:
        return None
hes_data_final['Language'] = hes_data_final['Comment'].apply(detect_language)

YouTube Dataset

**Data structure :**The YouTube dataset consists of various columns, each containing specific information related to YouTube videos and comments. These columns include the video title, channel name, comment date, comment content, number of likes, number of dislikes, author name, and number of replies. Each row represents a comment posted on a YouTube video, with details such as the video's title, the channel it belongs to, the date and time the comment was posted, the content of the comment itself, the number of likes and dislikes received, the author's username, and the number of replies the comment has generated

youtube_datasets_all = []
youtube_dataset1 = pd.read_csv("/content/drive/MyDrive/Datasets_Youtube/ramadan_morocco_comments_youtube (1).csv")
youtube_datasets_all.append(youtube_dataset1)
youtube_dataset2 = pd.read_csv("/content/drive/MyDrive/Datasets_Youtube/ramadan_morocco_comments_youtube (2).csv")
youtube_datasets_all.append(youtube_dataset2)
youtube_dataset3 = pd.read_csv("/content/drive/MyDrive/Datasets_Youtube/ramadan_morocco_comments_youtube (3).csv")
youtube_datasets_all.append(youtube_dataset3)
youtube_dataset4 = pd.read_csv("/content/drive/MyDrive/Datasets_Youtube/ramadan_morocco_comments_youtube (4).csv")
youtube_datasets_all.append(youtube_dataset4)
youtube_dataset5 = pd.read_csv("/content/drive/MyDrive/Datasets_Youtube/ramadan_morocco_comments_youtube (5).csv")
youtube_datasets_all.append(youtube_dataset5)
youtube_dataset6 = pd.read_csv("/content/drive/MyDrive/Datasets_Youtube/ramadan_morocco_comments_youtube (6).csv")
youtube_datasets_all.append(youtube_dataset6)
youtube_dataset7 = pd.read_csv("/content/drive/MyDrive/Datasets_Youtube/ramadan_morocco_comments_youtube (7).csv")
youtube_datasets_all.append(youtube_dataset7)
youtube_dataset8 = pd.read_csv("/content/drive/MyDrive/Datasets_Youtube/ramadan_morocco_comments_youtube (8).csv")
youtube_datasets_all.append(youtube_dataset8)
youtube_dataset9 = pd.read_csv("/content/drive/MyDrive/Datasets_Youtube/ramadan_morocco_comments_youtube (9).csv")
youtube_datasets_all.append(youtube_dataset9)
youtube_dataset10 = pd.read_csv("/content/drive/MyDrive/Datasets_Youtube/ramadan_morocco_comments_youtube (10).csv")
youtube_datasets_all.append(youtube_dataset10)
youtube_dataset11 = pd.read_csv("/content/drive/MyDrive/Datasets_Youtube/ramadan_morocco_comments_youtube (11).csv")
youtube_datasets_all.append(youtube_dataset11)
youtube_dataset12 = pd.read_csv("/content/drive/MyDrive/Datasets_Youtube/ramadan_morocco_comments_youtube (12).csv")
youtube_datasets_all.append(youtube_dataset12)
youtube_dataset13 = pd.read_csv("/content/drive/MyDrive/Datasets_Youtube/ramadan_morocco_comments_youtube (13).csv")
youtube_datasets_all.append(youtube_dataset13)
youtube_dataset14 = pd.read_csv("/content/drive/MyDrive/Datasets_Youtube/ramadan_morocco_comments_youtube (14).csv")
youtube_datasets_all.append(youtube_dataset14)
youtube_dataset15 = pd.read_csv("/content/drive/MyDrive/Datasets_Youtube/ramadan_morocco_comments_youtube.csv")
youtube_datasets_all.append(youtube_dataset15)

df_youtube = pd.concat(youtube_datasets_all, ignore_index=True)

Data cleaning : In the initial phase of cleaning the YouTube dataset, we began by addressing missing values and duplicate rows. Missing values, if left unhandled, can introduce bias and inaccuracies into our analysis. Therefore, we meticulously examined the dataset for any missing information and removed rows where essential data was absent. Additionally, we identified and eliminated duplicate rows to ensure the integrity and reliability of our dataset

df_youtube = df_youtube.dropna()
df_youtube = df_youtube.drop_duplicates()

After handling missing values and duplicates, we cleaned links and special characters from comments using regular expressions. This step aimed to enhance the quality of textual data by removing noise and irrelevant information, ensuring a clean and focused dataset for analysis.

import re

def clean_comment(comment):

    # Regular expression pattern to match URLs
    url_pattern = re.compile(r'http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\\(\\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+')

    # Replace URLs and special characters with an empty string
    cleaned_comment = re.sub(url_pattern, '', comment)

    return cleaned_comment

# Apply cleaning to comments
df_youtube['Comment'] = df_youtube['Comment'].apply(clean_comment)

After cleaning links and special characters, we detected the language of comments using the langdetect library. This step allowed us to identify the language of each comment, enhancing our understanding of the dataset's linguistic diversity and enabling more accurate analysis.

from langdetect import detect, DetectorFactory
DetectorFactory.seed = 0

# Create a function to detect the language of a comment
def detect_language(comment):
    try:
        lang = detect(comment)
        return lang
    except:
        return None

# Apply language detection to each comment and create a new column for language
df_youtube['Language'] = df_youtube['Comment'].apply(detect_language)

Drop rows with None values in the 'Language' column

df_youtube.dropna(subset=['Language'], inplace=True)

# Drop comments in English, French, and Spanish
df_youtube = df_youtube[(df_youtube['Language'] != 'en') & (df_youtube['Language'] != 'fr') & (df_youtube['Language'] != 'es') & (df_youtube['Language'] != 'ro') & (df_youtube['Language'] != 'cy') & (df_youtube['Language'] != 'no') & (df_youtube['Language'] != 'de') & (df_youtube['Language'] != 'et')& (df_youtube['Language'] != 'si')& (df_youtube['Language'] != 'af')& (df_youtube['Language'] != 'fi')& (df_youtube['Language'] != 'pt')& (df_youtube['Language'] != 'pl')& (df_youtube['Language'] != 'vi')& (df_youtube['Language'] != 'sv')& (df_youtube['Language'] != 'ca')& (df_youtube['Language'] != 'ti')& (df_youtube['Language'] != 'lv')& (df_youtube['Language'] != 'nl')& (df_youtube['Language'] != 'cs')& (df_youtube['Language'] != 'pt')& (df_youtube['Language'] != 'th')]

After detecting the language of comments, we added a new column called 'source', assigning the value 'YouTube' to each row

df_youtube['source'] = 'Youtube'

In the final step, we streamlined our dataset by selecting only the essential columns: Comment Date, Comment, Language, and Source. This focused approach ensures that we retain pertinent information for analysis while maintaining dataset clarity and efficiency

columns_to_drop = ['Video Title', 'Channel Name', 'Likes', 'Dislikes', 'Author', 'Replies']
df_youtube.drop(columns=columns_to_drop, inplace=True)

Facebook Dataset

Data structure : The Facebook dataset comprises several columns, including 'comment_text', 'comment_time', and 'Title'. The 'comment_text' column contains the textual content of comments posted on Facebook. The 'comment_time' column records the date and time of each comment's posting, and the 'Title' column likely denotes the title or headline associated with the Facebook post or content. Each row represents a comment posted on Facebook, with corresponding details such as comment text, posting time, and associated title.

facebook_df= pd.read_csv("/content/facebook_comments.csv")

Data cleaning : In the cleaning process, we determine the language of each comment using a language classification function. This step categorizes comments in the Facebook dataset as either Arabic or French based on the presence of specific language characters. It helps organize the data by language, facilitating subsequent analysis and processing tailored to each language category

import re

def classify_language(comment):
    """
    Classify the language of a comment as 'ar' (Darija-arabic) or 'fr' (darija-French).
    """
    # Regular expressions for Arabic and French characters
    arabic_pattern = re.compile(r'[\u0600-\u06FF\u0750-\u077F\u08A0-\u08FF]+')  # Arabic Unicode range
    english_pattern = re.compile(r'[\x00-\x7F\x80-\xFF]+')  # English Unicode range

    # Check if the comment contains Arabic or French characters
    if arabic_pattern.search(comment):
        return 'ar'
    elif english_pattern.search(comment):
        return 'fr'
    else:
        return None  # Return None if no Arabic or French characters are found

facebook_df['Language'] = facebook_df['comment_text'].apply(classify_language)

In the final step of cleaning, we add a new column labeled 'source' with the value 'Facebook'. Additionally, we streamline the dataset by selecting only the essential columns, excluding the 'Title' column. This focused approach ensures that the dataset retains pertinent information for analysis, enhancing clarity and efficiency.

facebook_df['source'] = 'Facebook'
facebook_df = facebook_df.drop(columns=['Title'])

NLP for data extraction and stop words removal

Machine Learning heavily relies on the quality of the data fed into it, and thus, data preprocessing plays a crucial role in ensuring the accuracy and efficiency of the model.

Text pre-processing is the process of preparing text data so that machines can use the same to perform tasks like analysis, predictions, etc.

There are many different steps in text pre-processing but in this article, we'll delve into the intricacies of preprocessing Arabic text data for sentiment analysis. From importing datasets to cleaning and preparing text for analysis, we'll explore each step in detail.

Importing Datasets:

To kickstart our preprocessing journey, we begin by importing our datasets. We've curated data from various sources including Facebook, Twitter, Hespress, and YouTube.

import pandas as pd
import os

#import the datasets
folder="C:/Users/sejja/Downloads/Compressed/cleaned_data/cleaned_data/"

df_facebook=pd.read_csv(folder+"facebook_clean.csv")

df_twitter=pd.read_csv(folder+"twitter_clean.csv")
df_hespress=pd.read_csv(folder+"hespress_clean.csv")
df_youtube=pd.read_csv(folder+"youtube_clean.csv")

df_facebook.columns,df_twitter.columns,df_hespress.columns,df_youtube.columns

Normalisation:

Normalisation ensures consistency and coherence in our data. Here, we align column names and select relevant columns from each dataset for further processing.

# Rename columns to match
df_twitter = df_twitter.rename(columns={'Content': 'Comment'})
df_twitter = df_twitter.rename(columns={'language': 'Language'})
df_youtube = df_youtube.rename(columns={'Comment Date': 'Date'})

# Select only the desired columns
df_facebook = df_facebook[['Comment', 'Date', 'Language', 'source']]
df_twitter = df_twitter[['Comment', 'Date', 'Language', 'source']]
df_hespress = df_hespress[['Comment', 'Date', 'Language', 'source']]
df_youtube = df_youtube[['Comment', 'Date', 'Language', 'source']]


# Concatenate the dataframes
df_merge = pd.concat([df_facebook, df_twitter, df_hespress, df_youtube], ignore_index=True)

Replacing Null Values:

Handling missing values is crucial for robust analysis. Here, we replace null values in the 'Language' column with 'ar' (Arabic) and drop rows with null values in the 'Date' column.

# Replace null values in the 'Language' column with 'ar'
df_merge['Language'].fillna('ar', inplace=True)

# Drop rows with null values in the 'Date' column
df_merge.dropna(subset=['Date'], inplace=True)

Deleting comments with less than 2 words

To maintain data quality, we filter out comments with less than two words.

df_merge = df_merge[df_merge['Comment'].str.split().str.len() >= 2]

Emoji replacement

In our text processing steps, we understand that emojis are important for showing feelings and emotions. So, we make sure to handle them properly. First, we create a list that matches each emoji with its meaning in words. Then, we create a function that goes through the text and changes each emoji to its word meaning using the list we made. This helps us better understand the text and the emotions it conveys.

# Create dictionary for emoji replacement
emoji_dict = dict(zip(df_emoji['emoji'], df_emoji['text']))

# Function to replace emojis with their meanings
def replace_emoji(text):
    for emoji, meaning in emoji_dict.items():
        text = text.replace(emoji, meaning)
    return text

Saving Data:

We save the preprocessed data for future analysis, including separate datasets for Arabic and French comments, and a merged dataset for comprehensive analysis.

datasetFr = df_merge[df_merge['Language'] == 'fr']
datasetAr = df_merge[df_merge['Language'] == 'ar']
datasetFr.to_csv("dataset_fr.csv")
datasetAr.to_csv("dataset_ar.csv")
df_merge.to_csv('merge_data.csv')

Preprocessing and Cleaning:

Now comes the heart of our preprocessing pipeline. We'll clean the text data by removing non-Arabic words, tokenizing, removing stopwords, and stemming.

So, let’s get started.

Removing non-arabic words

To ensure that only Arabic text remains for analysis, we utilize a regular expression pattern that matches any characters outside the Unicode range for Arabic script (\u0600 to \u06FF). This pattern effectively identifies and removes any non-Arabic characters, preserving the integrity of the text data for further processing.

Removing punctuation:

Punctuation marks, while essential for readability in natural language, often add noise to text analysis tasks. By employing another regular expression pattern targeting non-word characters (\w) and non-whitespace characters (\s), we systematically eliminate all punctuation from the text, streamlining the subsequent tokenization process.

What is tokenization:

Tokenization is the process of breaking down large blocks of text such as paragraphs and sentences into smaller, more manageable units.

What are stop words?

The words which are generally filtered out before processing a natural language are called stop words. These are actually the most common words in any language (like articles, prepositions, pronouns, conjunctions, etc) and does not add much information to the text. Examples of a few stop words in Arabic are
"كل" "لم" "لن" "له" "من" "هو" "هي" .

Why do we remove stop words?

Stop words are available in abundance in any human language. By removing these words, we remove the low-level information from our text in order to give more focus to the important information. In order words, we can say that the removal of such words does not show any negative consequences on the model we train for our task.

Removal of stop words definitely reduces the dataset size and thus reduces the training time due to the fewer number of tokens involved in the training.

What is stemming:

Stemming is the process of removing prefixes or suffixes from words to obtain their base form, known as the stem. For instance, words like “running,” “runner,” and “runs” share the same root “run.” Stemming helps consolidate words with similar meanings and reduces inflected words to a common form, aiding in tasks like text classification, sentiment analysis, and search engines.

Common Stemming Technique:

Snowball Stemmer: Also known as the Porter2 Stemmer, the Snowball Stemmer is an extension of the Porter algorithm with support for multiple languages. It employs a more systematic approach and can handle stemming tasks in languages beyond English, including French, German, and in our case Arabic.

Functions used to preprocess data

These functions serve as integral components of our data preprocessing pipeline, designed to clean and standardize textual data for subsequent analysis. By employing a combination of techniques such as removing non-Arabic words, eliminating punctuation, tokenizing text, removing stopwords, and applying stemming, we ensure that our data is refined and optimized for further analysis

# Function to clean and preprocess text without Stemming
def clean_text(text):
    # Remove non-Arabic words
    text = re.sub(r'[^\u0600-\u06FF\s]', '', text)
    # Remove punctuation
    text = re.sub(r'[^\w\s]', '', text)
    # Tokenization
    tokens = text.split()
    # Remove stopwords
    tokens = [word for word in tokens if word not in stopwords_darija]
    # Join tokens back into text
    clean_text = ' '.join(tokens)
    return clean_text

# with stemming

def preprocess_text(text):
    # Remove non-Arabic words
    text = re.sub(r'[^\u0600-\u06FF\s]', '', text)
        # Remove punctuation
    text = re.sub(r'[^\w\s]', '', text)
    # Tokenization
    tokens = word_tokenize(text.lower())
    # Remove stopwords
    tokens = [token for token in tokens if token not in stopwords_darija]
    # Stemming - using SnowballStemmer for Arabic languages
    stemmer = SnowballStemmer('arabic')
    tokens = [stemmer.stem(token) for token in tokens]
    return ' '.join(tokens)

Finally, we concatenate the preprocessed tokens back into a single string, using ' '.join(tokens), and return the resulting cleaned and normalized text from the function. This consolidated representation of the preprocessed text serves as the foundation for subsequent analysis and modeling tasks, enabling researchers and practitioners to derive meaningful insights from Arabic text data with confidence and accuracy.

• Below is a visual representation of preprocessing execution, showcasing the execution of the preprocessing functions discussed above. This image provides a glimpse into the transformation of raw textual data into a refined and standardized format, illustrating the effectiveness of the cleaning, tokenization, and stemming processes in preparing the data for subsequent analysis.

In Summary:

The steps we've taken to prepare our text data are crucial for making sense of it in natural language processing. By cleaning, organizing, and simplifying the text through various techniques, we've made it more understandable and ready for analysis. This process helps us remove unnecessary information, standardize the text, and focus on what truly matters. With our data in better shape, we're now well-equipped to delve deeper into analysis tasks such as understanding sentiment, categorizing text, and finding relevant information. Overall, these preprocessing steps form a solid foundation for extracting valuable insights from text data, enhancing the accuracy and effectiveness of our analysis.

Unveiling Insights: Exploratory Data Analysis on Ramadan Comments in Moroccan Darija

As the digital sphere continues to evolve, social media platforms serve as a rich source of public opinion and sentiment. In this blog, we delve into the world of Ramadan comments, examining the chatter before and after Iftar across popular platforms like Facebook, Twitter, Hespress, and YouTube. Our focus lies on deciphering patterns, trends, and unique characteristics within the comments, all in the vibrant language of Moroccan Darija.

In this article, you will learn how to:

• Conduct word frequency analysis to identify predominant themes in Ramadan comments.

• Explore temporal trends, revealing higher activity before Iftar.

• Analyze language distribution, with Arabic comments prevailing.

• Apply topic modeling to categorize comments into distinct themes.

• Perform time series analysis to understand comment dynamics before and after Iftar.

This simplified tutorial will guide you through each phase of EDA applications, with a special focus on interpreting and visualizing the results.

You can find the entire code in the notebook below:

Get ready :)

Introduction to EDA

Exploratory Data Analysis (EDA) serves as the cornerstone of data exploration, offering a systematic approach to uncovering patterns, trends, and insights within datasets. In this section, we delve into the theoretical underpinnings of EDA, laying the groundwork for our journey through the Ramadan comments dataset in Moroccan Darija.

Essence of EDA

At its core, EDA embodies a philosophy of curiosity and discovery, empowering data practitioners to glean meaningful insights from raw datasets. Unlike formal statistical methods, which often require predefined hypotheses, EDA embraces a more flexible and intuitive approach, allowing analysts to let the data speak for itself.

Key Principles of EDA:

1. Visualization: Visualization lies at the heart of EDA, enabling analysts to transform raw data into insightful plots, charts, and graphs. By visually inspecting the distribution, relationships, and anomalies within the data, analysts can uncover hidden patterns and outliers that may elude traditional statistical methods.

2. Descriptive Statistics: Descriptive statistics provide a snapshot of the dataset's central tendencies, variability, and distribution. Metrics such as mean, median, standard deviation, and percentiles offer valuable insights into the shape and characteristics of the data, guiding subsequent analysis and interpretation.

3. Data Cleaning and Preprocessing: Before embarking on exploratory analysis, it is essential to ensure the cleanliness and integrity of the dataset. Data cleaning involves identifying and addressing missing values, outliers, and inconsistencies that may distort the analysis. Additionally, preprocessing steps such as normalization and transformation may be employed to enhance the quality and interpretability of the data.

4. Pattern Recognition: EDA involves the systematic identification of patterns, trends, and relationships within the dataset. By applying statistical techniques such as correlation analysis, clustering, and dimensionality reduction, analysts can uncover meaningful structures and associations that underlie the data.

5. Iteration and Iterative Refinement: EDA is an iterative process, wherein analysts continuously refine their analysis based on emerging insights and hypotheses. By iteratively exploring, visualizing, and interpreting the data, analysts can gradually refine their understanding of the dataset and extract deeper insights.

Understanding the Dataset

Before embarking on our exploratory journey, let's grasp the essence of our dataset. It comprises comments captured during Ramadan, spanning the moments preceding and following Iftar, the evening meal that breaks the fast. These comments emanate from diverse sources, reflecting the sentiments, emotions, and discussions prevalent during this sacred time.

Exploring Patterns and Trends

Word Frequency Analysis

Word Frequency Analysis is a powerful technique used to extract meaningful insights from text data by quantifying the frequency of occurrence of individual words or phrases within a corpus. In this section, we apply Word Frequency Analysis to our Ramadan comments dataset in Moroccan Darija, aiming to uncover the most prevalent terms and themes within the discourse.

def display_frequency_plot(df, column, stopwords):
     reshaped_text = " ".join(arabic_reshaper.reshape(t) for t in df[column].dropna())

    plt.figure(figsize=(20, 10))

    counts = Counter(reshaped_text.split())
    counts = {get_display(k): v for k, v in counts.items() if k not in stopwords}
    counts = dict(sorted(counts.items(), key=lambda x: x[1], reverse=True)[:20])
    palette = sns.color_palette("crest_r", n_colors=len(counts))
    palette = dict(zip(counts.keys(), palette))
    sns.barplot(y=list(counts.keys()), x=list(counts.values()), palette=palette)

    plt.title("Frequency Plot")
    plt.show()

Interpretation of Results:

The Word Frequency Analysis reveals that the most frequent words in our dataset are predominantly religious, which can be attributed to the sacred nature of Ramadan. Terms like "الله" (Allah), "رمضان" (Ramadan), and "اللهم" (O Allah) dominate the discourse, reflecting the deep significance of faith and spirituality during this holy month. This observation underscores the cultural and societal norms surrounding Ramadan, where discussions often revolve around religious observance and spiritual reflection, shaping the digital discourse in meaningful ways.

Temporal Trends

In the realm of digital communication, understanding temporal patterns is crucial for deciphering trends and behaviors within online communities. In the context of Ramadan, a period marked by spiritual reflection, communal gatherings, and fasting, the temporal dynamics of online engagement hold particular significance.

Let's analyse first the temporal distribution of comments reveals interesting patterns, particularly in terms of peak activity during specific hours of the day.

data['Hour'] = data['Date'].dt.hour

commentaires_par_heure = data.groupby('Hour').size()

plt.figure(figsize=(10, 6))
commentaires_par_heure.plot(kind='bar', color='blue')
plt.title('Nombre de commentaires par heure de la journée')
plt.xlabel('Heure de la journée')
plt.ylabel('Nombre de commentaires')
plt.xticks(rotation=0)
plt.grid(axis='y', linestyle='--', alpha=0.7)
plt.tight_layout()
plt.show()

These findings highlight distinct peaks in commenting activity throughout the day, with the afternoon hours exhibiting the highest levels of engagement. Hour 15:00 emerges as the period of greatest activity, suggesting a concentration of discussions and interactions during this time frame. Conversely, late evening hours, such as 22:00 and 21:00, also witness significant participation, albeit to a slightly lesser extent.

Time Series Analysis

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

data['Date'] = pd.to_datetime(data['Date'], format='%Y-%m-%d %H:%M:%S', errors='coerce')
data.dropna(subset=['Date'], inplace=True)

def time_series_analysis_by_source():
    fig, axes = plt.subplots(len(data['source'].unique()), 2, figsize=(15, 6 * len(data['source'].unique())))

    for i, source in enumerate(data['source'].unique()):
        source_data = data[data['source'] == source]
        source_data['Part_of_Day'] = pd.cut(source_data['Date'].dt.hour, bins=[0, 6, 12, 18, 24],
                                            labels=['Night', 'Morning', 'Afternoon', 'Evening'])
        sns.countplot(x='Part_of_Day', data=source_data, ax=axes[i][0])
        axes[i][0].set_title(f'Distribution of Comments by Part of the Day for {source}')
        axes[i][0].set_xlabel('Part of the Day')
        axes[i][0].set_ylabel('Number of Comments')

        before_iftar = data[(data['Date'].dt.hour < 18) | ((data['Date'].dt.hour == 18) & (data['Date'].dt.minute < 30))].shape[0]
        after_iftar = data[(data['Date'].dt.hour >= 18) | ((data['Date'].dt.hour == 18) & (data['Date'].dt.minute >= 30))].shape[0]

        sns.barplot(x=['Before Iftar', 'After Iftar'], y=[before_iftar, after_iftar], ax=axes[i][1])
        axes[i][1].set_title(f'Total Number of Comments before and after Iftar for {source}')
        axes[i][1].set_xlabel('Part of the Day')
        axes[i][1].set_ylabel('Total Number of Comments')

    plt.tight_layout()
    plt.show()

def time_series_analysis_by_topic():
    fig, axes = plt.subplots(len(data['topic'].unique()), 2, figsize=(15, 6 * len(data['topic'].unique())))

    for i, topic in enumerate(data['topic'].unique()):
        topic_data = data[data['topic'] == topic]
        topic_data['Part_of_Day'] = pd.cut(topic_data['Date'].dt.hour, bins=[0, 6, 12, 18, 24],
                                            labels=['Night', 'Morning', 'Afternoon', 'Evening'])
        sns.countplot(x='Part_of_Day', data=topic_data, ax=axes[i][0])
        axes[i][0].set_title(f'Distribution of Comments by Part of the Day for Topic: {topic}')
        axes[i][0].set_xlabel('Part of the Day')
        axes[i][0].set_ylabel('Number of Comments')

        before_iftar = data[(data['Date'].dt.hour < 18) | ((data['Date'].dt.hour == 18) & (data['Date'].dt.minute < 30))].shape[0]
        after_iftar = data[(data['Date'].dt.hour >= 18) | ((data['Date'].dt.hour == 18) & (data['Date'].dt.minute >= 30))].shape[0]

        sns.barplot(x=['Before Iftar', 'After Iftar'], y=[before_iftar, after_iftar], ax=axes[i][1])
        axes[i][1].set_title(f'Total Number of Comments before and after Iftar for Topic: {topic}')
        axes[i][1].set_xlabel('Part of the Day')
        axes[i][1].set_ylabel('Total Number of Comments')

    plt.tight_layout()
    plt.show()

Increased Pre-Iftar Engagement: Before Iftar, there is a notable uptick in comment activity, indicating heightened engagement on social media platforms. This surge in activity can be attributed to various factors:

• Anticipation and Preparation: Individuals actively discuss meal preparations, share recipes, and express excitement for the impending breaking of the fast. Additionally, conversations about Ramadan traditions and cultural practices contribute to the vibrant pre-Iftar discourse.

• Real-Time Sharing: The period preceding Iftar witnesses a surge in real-time sharing of fasting experiences and spiritual reflections. Users turn to social media to express personal anecdotes, seek communal support, and engage in collective reflection on the significance of Ramadan.

Conversely, there is a discernible decrease in comment activity after Iftar. Several factors may underlie this decline:

Post-Iftar Social Engagement: Following the breaking of the fast, individuals prioritize spending time with family and friends, participating in communal prayers, and enjoying post-Iftar meals and social gatherings. Consequently, there is a natural diversion of attention away from social media platforms, leading to reduced comment volumes.

Dynamic Nature of Online Behavior: The fluctuation in online engagement before and after Iftar underscores the dynamic interplay between cultural traditions, religious observance, and digital interaction during Ramadan. These patterns highlight the evolving nature of user behavior and preferences within the context of this sacred month.

Language Distribution

Examining the distribution of languages within the dataset reveals a notable disparity between Arabic and French comments, with Arabic comments significantly outnumbering French comments. This observation underscores the predominance of Arabic as the primary language of communication and expression within the context of Ramadan discussions.

The overwhelming presence of Arabic comments reflects the cultural and linguistic dynamics inherent in conversations surrounding Ramadan, particularly in regions where Arabic is the predominant language of communication. Arabic serves as the medium through which individuals articulate their thoughts, sentiments, and reflections on the significance of Ramadan, fostering a sense of shared cultural identity and belonging among participants.

While French comments may represent a minority within the dataset, their presence underscores the linguistic diversity and multiculturalism inherent in online discourse surrounding Ramadan. These comments may originate from individuals with varying linguistic backgrounds, contributing to the richness and diversity of perspectives within the digital conversation.

Topic Modeling

Topic modeling is a powerful method for uncovering recurring themes and patterns within textual data. In our analysis of Ramadan comments, we use topic modeling to reveal the underlying topics prevalent in the discourse. By identifying these themes, we gain valuable insights into the diverse narratives and interests shaping conversations during this sacred month.

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation

# Convert comments to a list
comments_list = data['clean_comment'].tolist()

# Create a CountVectorizer to convert comments into a matrix of token counts
vectorizer = CountVectorizer(max_df=0.95, min_df=2, stop_words=None)
X = vectorizer.fit_transform(comments_list)

# Specify the number of topics
n_topics = 3

# Run LDA
lda_model = LatentDirichletAllocation(n_components=n_topics, max_iter=10, learning_method='online', random_state=42)
lda_Z = lda_model.fit_transform(X)

# Get the feature names (words)
feature_names = vectorizer.get_feature_names_out()

# Create a dictionary to store the top words for each topic
topics_dict = {}
n_top_words = 20
for topic_idx, topic in enumerate(lda_model.components_):
    topic_name = f"Topic_{topic_idx}"
    topic_words = [feature_names[i] for i in topic.argsort()[:-n_top_words - 1:-1]]
    topics_dict[topic_name] = topic_words

# Print the topic names and their respective top words
for topic_name, topic_words in topics_dict.items():
    print(f"{topic_name}: {', '.join(topic_words)}")

Utilizing topic modeling techniques, we have identified three distinct topics within the dataset:

Topic_0: This topic revolves around religious themes, with keywords such as "الله" (Allah), "رمضان" (Ramadan), and "اللهم" (O Allah) indicating discussions centered on faith, supplication, and spiritual reflection. Other terms like "فلسطين" (Palestine) suggest engagement with social and humanitarian issues within a religious context.

Topic_1: The keywords in this topic pertain to entertainment and media content, with terms like "مسلسل" (series), "فيلم" (film), and "حلقة" (episode) indicating discussions related to television shows, movies, and popular culture. This topic reflects a divergence from religious discourse, focusing instead on leisure and entertainment activities.

Topic_2: This topic encompasses a mix of religious and cultural references, with keywords such as "المغرب" (Morocco), "محمد" (Mohammed), and "صلى" (prayer) suggesting discussions related to Islamic teachings, Moroccan culture, and expressions of gratitude and blessings.

Extracting Top Bigrams for Each Topic Using CountVectorizer

from sklearn.feature_extraction.text import CountVectorizer

# Define a function to extract top bigrams for each topic
def extract_top_bigrams(topic):
    # Filter comments based on the topic
    topic_comments = data[data['topic'] == topic]['clean_comment'].tolist()

    # Create a CountVectorizer to extract bigrams
    vectorizer = CountVectorizer(ngram_range=(2, 5), max_features=10000)
    X = vectorizer.fit_transform(topic_comments)

    # Get feature names (bigrams)
    feature_names = vectorizer.get_feature_names_out()

    # Get counts of each bigram
    bigram_counts = X.sum(axis=0)

    # Create a dictionary to store bigrams and their counts
    bigram_dict = {bigram: count for bigram, count in zip(feature_names, bigram_counts.A1)}

    # Get the top 20 most common bigrams
    top_20_bigrams = sorted(bigram_dict.items(), key=lambda x: x[1], reverse=True)[:20]

    # Print header
    print(f"{'='*40}\nTop 20 most common bigrams for {topic}:\n{'='*40}")

    # Print the top 20 most common bigrams
    for i, (bigram, count) in enumerate(top_20_bigrams, 1):
        print(f"{i}. {bigram}: {count} occurrences")

# Iterate over topics and extract top bigrams for each topic
topics = data['topic'].unique()
for topic in topics:
    extract_top_bigrams(topic)

The bigrams extracted for the Entertainment topic predominantly consist of phrases related to television shows, movies, and online content. Phrases like "احسن مسلسل" (best series), "مسلسل زوين" (nice series), and "دنيا بوطازوت" (Dounia Batma, a Moroccan singer and actress) indicate discussions about specific TV series and personalities. Additionally, expressions such as "شي لايكات" (some likes) and "لايكات نحسو" (likes we count) suggest engagement with social media metrics and audience interaction. The prevalence of laughter-related phrases like "ههههههههههه" (hahaha) underscores the informal and lighthearted nature of the discussions within this topic.

In contrast, the bigrams extracted for the Religion and Ramadan topic are predominantly religious expressions and blessings associated with Ramadan. Phrases like "تبارك الله" (blessings of Allah), "رمضان كريم" (blessed Ramadan), and "اللهم صل" (O Allah, bless) reflect the reverence and piety inherent in Ramadan-related discourse. Expressions such as "جزاكم الله" (may Allah reward you) and "ماشاء الله" (as Allah wills) convey expressions of gratitude and acknowledgment of divine blessings. The repetition of phrases like "الله الله الله" (Allah Allah Allah) emphasizes the centrality of God in the conversations, reflecting a deep spiritual connection and devotion among participants.

Comment Length Analysis

In exploring the dynamics of online discourse, descriptive statistics offer valuable insights into the characteristics and patterns of communication within different thematic categories. In this context, we analyze descriptive statistics for comments categorized under two distinct topics: Entertainment and Religion/Ramadan. By examining mean comment length, mean word count, and mean number of characters, we gain a deeper understanding of the communication styles and content preferences within each topic.

import matplotlib.pyplot as plt
import seaborn as sns

def length_analysis(topic, col):
    topic_comments = data[data[col] == topic]['clean_comment']
    comment_lengths = topic_comments.apply(lambda x: len(x))
    word_counts = topic_comments.apply(lambda x: len(x.split()))
    mean_length = comment_lengths.mean()
    mean_word_count = word_counts.mean()
    mean_characters = topic_comments.apply(len).mean()

    # Print descriptive statistics
    print(f"\n{'='*40}\nDescriptive Statistics for {topic}:\n{'='*40}")
    print(f"Mean Comment Length: {mean_length:.2f} characters")
    print(f"Mean Word Count: {mean_word_count:.2f}")
    print(f"Mean Number of Characters: {mean_characters:.2f} characters")

    # Create visualization
    fig, axes = plt.subplots(1, 3, figsize=(15, 5))

    sns.histplot(comment_lengths, kde=True, ax=axes[0])
    axes[0].set_title('Comment Length Distribution')
    axes[0].set_xlabel('Comment Length')
    axes[0].set_ylabel('Frequency')

    sns.histplot(word_counts, kde=True, ax=axes[1])
    axes[1].set_title('Word Count Distribution')
    axes[1].set_xlabel('Word Count')
    axes[1].set_ylabel('Frequency')

    sns.histplot(topic_comments.apply(len), kde=True, ax=axes[2])
    axes[2].set_title('Character Count Distribution')
    axes[2].set_xlabel('Number of Characters')
    axes[2].set_ylabel('Frequency')

    plt.tight_layout()
    plt.show()

# Iterate over topics and perform length analysis for each topic
topics = data['topic'].unique()
for topic in topics:
    length_analysis(topic, 'topic')

For comments categorized under the Entertainment topic, the descriptive statistics reveal a mean comment length of 44.70 characters. On average, each comment consists of approximately 7.72 words. The mean number of characters per comment aligns with the mean comment length, indicating a relatively concise style of communication within this topic. This suggests that discussions related to entertainment tend to be succinct and to the point, reflecting the informal and casual nature of the conversations.

In contrast, comments categorized under the Religion and Ramadan topic exhibit a longer mean comment length of 68.84 characters. On average, each comment in this category comprises approximately 11.53 words. The higher mean comment length and word count suggest a more elaborate and detailed style of expression within discussions related to religion and Ramadan. Participants in these discussions may engage in more in-depth reflections, prayers, and expressions of devotion, contributing to the longer and more nuanced comments observed in this category.

Analyzing Comment Length and Word Count by Source Platform

• Facebook

  Comments originating from Facebook have a mean length of 59.61 characters and a mean word count of 9.77. This suggests a moderately concise communication style, with comments typically comprising short sentences or phrases. The relatively lower mean comment length compared to other platforms may indicate a tendency for more succinct interactions on Facebook.

  

• Twitter

  Twitter comments exhibit a slightly longer mean length of 63.06 characters and a mean word count of 10.52. While still concise, Twitter users appear to engage in slightly more extensive communication compared to Facebook. This may be attributed to Twitter's character limit and the need for users to convey their message within a constrained space.

  

• Hespress

  In contrast, comments from Hespress display a significantly longer mean length of 156.21 characters and a mean word count of 24.26. These statistics indicate a more verbose and detailed communication style on Hespress, with users often expressing complex thoughts or opinions in their comments. The higher mean comment length suggests that discussions on Hespress may be more in-depth and comprehensive compared to other platforms.

  

• YouTube

  Comments on YouTube have a mean length of 54.66 characters and a mean word count of 9.32. Similar to Facebook, YouTube comments tend to be relatively concise, with users typically conveying their thoughts or reactions in short messages. The lower mean comment length may reflect the nature of interactions on YouTube, where comments are often brief and focused on immediate reactions to the content.

In conclusion, this comprehensive exploration of comment data during Ramadan sheds light on various aspects of online discourse during this sacred month. Through techniques such as word frequency analysis, temporal trend exploration, language distribution analysis, topic modeling, and time series analysis, we've gained valuable insights into the patterns, themes, and dynamics of comments before and after Iftar. This analysis not only enhances our understanding of online engagement during Ramadan but also underscores the importance of Exploratory Data Analysis (EDA) in uncovering meaningful insights from complex datasets. As we continue to delve deeper into data analysis, it's crucial to leverage these techniques to derive actionable insights that can inform decision-making and deepen our understanding of societal trends and behaviors.

Machine Learning and Deep Learning models

In our project, we harness the power of machine learning (ML) and deep learning (DL) to unlock valuable insights from Moroccan Darija comments and tweets. Our approach involves leveraging a variety of ML and DL models to tackle different aspects of the data analysis process.

Clustering Models for Topic Segmentation

To uncover underlying themes and topics within the dataset, we utilize clustering models. These models group similar comments and tweets together based on their content, allowing us to identify distinct topics of discussion within the Moroccan online community. By applying clustering techniques, we gain a deeper understanding of the diverse range of subjects that are relevant to Moroccan users.

Next Word Prediction Model

Another crucial aspect of our project is the development of a next word prediction model. This model predicts the most probable word to follow a given sequence of words, taking into account the context of the conversation. By accurately predicting the next word, we enhance the coherence and fluency of generated text, improving the overall quality of our analysis and insights.

Models Available on Hugging Face via the Link Below

DARIJABERT: Sentiment Prediction Model

One of the highlights of our project is the implementation of DARIJABERT, a specialized sentiment prediction model trained on Moroccan Darija data. Inspired by state-of-the-art language models like BERT, DARIJABERT excels in understanding and predicting sentiment in Moroccan Darija comments and tweets. By leveraging deep learning techniques, DARIJABERT enables us to accurately assess the sentiment expressed in online discussions, providing valuable insights into the attitudes and opinions of Moroccan users.

Conclusion

Through the application of ML and DL models, we are able to delve deep into the world of Moroccan online discourse, uncovering hidden patterns, sentiments, and topics of interest. These advanced techniques empower us to extract meaningful insights from vast amounts of data, ultimately enhancing our understanding of Moroccan culture, society, and online behavior.

Conclusion

As we conclude this chapter, we are reminded of the transformative potential of collaborative endeavors like those championed by the MDS community. By harnessing the power of data and technology, we not only gain insights into societal trends and cultural expressions but also pave the way for informed decision-making, community engagement, and positive social change.

As we look towards the future, let us continue to embrace the spirit of collaboration, curiosity, and inclusivity that defines the MDS community. Together, we will continue to explore, innovate, and inspire, shaping a brighter tomorrow through the lens of data science and community-driven initiatives.

Explore a preview of our project : https://moroccansentimentsanalysis.com

This article is your gateway to understanding and utilizing news data effectively. Here, we will walk you through a straightforward yet effective approach to gather, organize, analyze, and use data from Moroccan news websites. We will use easy-to-understand methods involving popular Python tools like Streamlit, Pandas, Selenium, along with Google Drive Handle and python-dotenv. Our aim is to make this journey from collecting data to applying it as smooth as possible for you.

Understanding Data Scraping

Data scraping is the process of collecting information from websites. This is commonly done to gather data from various sources on the internet for analysis, research, or storage.

Data scraping can be especially useful for keeping track of changes on websites, gathering large amounts of data quickly, and automating repetitive tasks of collecting information.

However, it's important to remember that while data scraping is a powerful tool, it should be used responsibly and ethically. This means respecting website terms of use, considering data privacy laws, and not overloading websites with too many requests at once.

Setting Up

To get started with our project, you'll need to set up your computer with some specific tools. Here's a list of what you need and how to install them:

1. Streamlit: This tool helps us build and share web applications easily. Install it using the command: pip install streamlit.

2. Pandas: A library that makes handling data easier. Install it with pip install pandas.

3. Selenium (version 4.0.0 to less than 5.0.0): It's crucial for web scraping. Install the correct version using pip install selenium==4.*.

4. PyDrive: This tool will help us work with Google Drive files. Install it using pip install PyDrive.

5. python-dotenv: This library will help us manage environment variables securely. Install it with pip install python-dotenv.

6. WebDriver Manager: It helps in managing the browser drivers needed for Selenium. Install it using pip install webdriver_manager.

7. BeautifulSoup4: A library that makes it easy to scrape information from web pages. Install it with pip install beautifulsoup4.

With these tools installed, you'll be ready to start working on your data scraping and analysis project!

Data Cleaning with Pandas

After successfully scraping data from websites, the next crucial step is cleaning this data. This means making sure the data is organized, free from errors, and ready for analysis. For this, we use Pandas, a powerful Python library that simplifies the process of data manipulation.

Getting Started with Pandas

1. Loading Data: Begin by loading the scraped data into Pandas for easy manipulation.

2. Removing Duplicates: Identify and remove any duplicate entries to ensure data quality.

3. Handling Missing Values: Find and address any missing or incomplete information in the dataset.

4. Formatting Data: Adjust data formats as needed for consistency and easier analysis.

    import pandas as pd
    # Load the dataset
    df = pd.read_csv('scraped_articles.csv')
    # Remove duplicates
    df.drop_duplicates(subset='article_title', inplace=True)
    # Remove any rows with missing values
    df.dropna(inplace=True)
    df.to_csv('cleaned_articles.csv', index=False)
   

Integrating Google Drive API for Data Storage and Sharing:

In our project, we leverage the Google Drive API to efficiently store and share the data we've collected. This approach not only provides a secure and scalable storage solution but also simplifies the process of making our data accessible to others. Here's how we do it:

Setting Up the Environment

1. Importing Libraries: We start by importing necessary libraries including pydrive.auth, GoogleDrive, and oauth2client.client.

2. Environment Variables: Using load_dotenv from the dotenv package, we load our Google API credentials stored in environment variables for security. This includes CLIENT_ID, CLIENT_SECRET, and REFRESH_TOKEN.

Authenticating with Google Drive

• Authentication Function: We define a function authenticate_google_drive which sets up the authentication using OAuth2 credentials. This ensures a secure connection to Google Drive.

Uploading Files to Google Drive

• Upload Function: The upload_file_to_drive function takes in the drive object and the file path to upload the file to Google Drive. We also handle the case where the file might already exist on Google Drive, updating it instead of uploading a duplicate.

• Error Handling: The function includes error handling to manage any issues during the upload process.

Generating Downloadable Links

• Download Link Function: The get_drive_download_link function is used to generate a direct download link for the files uploaded. This function sets the necessary permissions on the file to make it accessible to anyone with the link.

Practical Example

from dotenv import load_dotenv
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from oauth2client.client import OAuth2Credentials
import os

load_dotenv()

CLIENT_ID = os.getenv('CLIENT_ID')
CLIENT_SECRET = os.getenv('CLIENT_SECRET')
REFRESH_TOKEN = os.getenv('REFRESH_TOKEN')
REDIRECT_URI = os.getenv('REDIRECT_URIS').split(',')[0]  # Access the first URI

    def authenticate_google_drive():
        gauth = GoogleAuth()
        gauth.credentials = OAuth2Credentials(None, CLIENT_ID, CLIENT_SECRET,REFRESH_TOKEN, None,
                                             "https://accounts.google.com/o/oauth2/token", None, "web")
        drive = GoogleDrive(gauth)
        return drive

    drive = authenticate_google_drive()

    def upload_file_to_drive(drive, file_path, folder_id=None):
        if not os.path.exists(file_path):
            print(f"Cannot upload, file does not exist at path: {file_path}")
            return None

        try:
            file_metadata = {'title': os.path.basename(file_path)}
            if folder_id:
                file_metadata['parents'] = [{'id': folder_id}]

            upload_file = drive.CreateFile(file_metadata)

            # Check if the file already exists on Google Drive
            existing_files = drive.ListFile({'q': f"title='{upload_file['title']}'"}).GetList()
            if existing_files:
                # File with the same name already exists, update the existing file
                upload_file = existing_files[0]
                print(f"File already exists on Drive. Updating file with ID: {upload_file['id']}")
            else:
                print("Uploading a new file to Drive.")

            upload_file.SetContentFile(file_path)
            upload_file.Upload()
            print(f"File uploaded successfully. File ID: {upload_file['id']}")
            return upload_file['id']
        except Exception as e:
            print(f"An error occurred during file upload: {e}")
            return None


    def get_drive_download_link(drive, file_id):
        try:
            file = drive.CreateFile({'id': file_id})
            file.Upload() # Make sure the file exists on Drive
            file.InsertPermission({
                'type': 'anyone',
                'value': 'anyone',
                'role': 'reader'})
            return "https://drive.google.com/uc?export=download&id=" + file_id
        except Exception as e:
            print(f"Error fetching download link: {e}")
            return None

Conclusion

This integration of the Google Drive API provides a robust and secure method for storing and sharing large datasets. By automating the upload and link generation process, we significantly enhance the accessibility and usability of our data.

Streamlit Web Interface

Our Streamlit application serves as a central hub for aggregating news from various Moroccan websites. It's designed to be intuitive and user-friendly, enabling users to select news sources, languages, and categories for scraping.

Key Features

1. Dynamic Configuration: Users can choose websites and specific categories from a dynamically loaded configuration (config.json). This allows for a customizable scraping experience.

2. Language and Category Selection: For websites offering content in multiple languages, users can select their preferred language. Additionally, users can pick specific categories of news they are interested in.

3. Control Over Data Collection: Through a simple interface, users can specify the number of articles to scrape.

4. Initiating Scraping: A 'Start Scraping' button triggers the scraping process, with a progress bar indicating the ongoing operation.

5. Real-time Updates and Data Display: As data is scraped and uploaded, users receive real-time updates. Each successful scrape results in a download link for the data and a display of the scraped data in a tabular format within the application.

6. Google Drive Integration: Scraped data files are uploaded to Google Drive, and direct download links are provided within the Streamlit interface for easy access.

7. Error Handling: The application includes error handling for issues like failed file uploads or unsuccessful scrapes, ensuring a smooth user experience.

    import streamlit as st
    import pandas as pd
    import json
    import importlib
    from selenium import webdriver
    from selenium.webdriver.chrome.options import Options as ChromeOptions
    import google_drive_handle as gdrive
    from dotenv import load_dotenv
    import os
   
    # Load config.json
    with open('config.json') as f:
        config = json.load(f)
   
    # Set up Chrome WebDriver with options
    options = ChromeOptions()
    options.add_argument('--headless')
    options.add_argument('--no-sandbox')
    options.add_argument('--disable-dev-shm-usage')
    options.add_argument('log-level=3')
    # Initialize the Chrome WebDriver
    wd = webdriver.Chrome(options=options)
    drive = gdrive.authenticate_google_drive()
    processed_files = set()
    st.markdown(
        """
        <style>
            .centered {
                display: flex;
                align-items: center;
                justify-content: center;
                text-align: center;
            }
        </style>
        """,
        unsafe_allow_html=True
    )
   
    st.markdown("<h1 class='centered'>Moroccan News Aggregator</h1>", unsafe_allow_html=True)
   
    selected_websites = {}
    selected_categories = {}
   
    def save_file_id_mapping(file_id_mapping):
        with open("file_id_mapping.json", "w") as file:
            json.dump(file_id_mapping, file)
   
    def load_file_id_mapping():
        try:
            with open("file_id_mapping.json", "r") as file:
                return json.load(file)
        except FileNotFoundError:
            return {}  # Return an empty dictionary if the file doesn't exist
   
    file_id_mapping = load_file_id_mapping()
   
    selected_websites = {}
   
    for website, details in config.items():
        if st.checkbox(website, key=website):
            # Language selection
            languages = details.get("languages", {})
            if languages and len(languages) > 1:
                language = st.selectbox(f'Choose language for {website}', list(languages.keys()), key=f'lang_{website}')
                selected_websites[website] = f"{website}_{language}"  # like: hespress_en
            else:
                selected_websites[website] = website  # like: akhbarona
   
            # Category selection
            categories = languages.get(language, {})
            if categories:
                categories = st.multiselect(f'Select categories for {website}', list(categories.keys()), key=f'{website}_categories')
                selected_categories[website] = categories
   
    # Number of articles input
    num_articles = st.number_input('Number of Articles', min_value=1, max_value=10000, step=1)
   
    # Start scraping button
    if st.button('Start Scraping'):
        with st.spinner('Scraping in progress...'):
            progress_bar = st.progress(0)
            total_tasks = sum(len(categories) for categories in selected_categories.values())
            completed_tasks = 0
            for website, module_name in selected_websites.items():
                scraper_module = importlib.import_module(module_name)
                for category in selected_categories.get(website, []):
                    category_url = config[website]['languages'][language][category]
                    if 'category_name' in config[website]:
                        category_name = config[website]['category_name'].get(category, 'default_category_name')
                    file_path = scraper_module.scrape_category(category_url, num_articles)
   
                    if file_path:
                        if file_path not in file_id_mapping:
                            file_id = gdrive.upload_file_to_drive(drive, file_path)
                            print(f"Uploading file: {file_path}, File ID: {file_id}")
                            file_id_mapping[file_path] = file_id
                            save_file_id_mapping(file_id_mapping)
                        else:
                            file_id = file_id_mapping[file_path]
                            print(f"File already uploaded. Using existing File ID: {file_id}")
   
                        if file_id:
                            download_link = gdrive.get_drive_download_link(drive, file_id)
                            if download_link:
                                st.markdown(f"[Download {website} - {category} data]({download_link})", unsafe_allow_html=True)
   
                                df = pd.read_csv(file_path)
                                st.write(f"{website} - {category} Data:")
                                st.dataframe(df)
                            else:
                                st.error(f"Failed to retrieve download link for file ID: {file_id}")
                        else:
                            st.error(f"Failed to upload file for {website} - {category}")
                    else:
                        st.error(f"File not created for {website} - {category}")
   
            st.success('Scraping Completed!')
   

This Streamlit web interface stands as a testament to the power of Python in creating efficient, user-friendly tools for data aggregation and management. It simplifies the complex process of data collection, storage, and sharing, making it accessible even to those with minimal technical background.

Dynamic Configuration with 'config.json'

Our Streamlit application is designed with agility and future expansion in mind. By incorporating a config.json file, we've created a flexible framework that allows for easy addition and modification of news sources.

The Role of config.json

• Flexible Source Management: The config.json file holds the details of various news websites, their available languages, and specific category URLs. This setup enables us to easily add new sources or modify existing ones without altering the core code of the application.

• Language and Category Customization: For each news website, multiple languages and categories are defined. Users can select their preferred language and categories, making the data scraping process highly customizable.

Implementation in Streamlit

• Loading Configuration: At the start of the application, config.json is loaded to dynamically populate the website choices, along with their respective languages and categories.

• User Interactions: Users interact with checkboxes and dropdowns generated based on the configuration. They can select websites, languages, and specific categories for scraping.

• Scalability: Adding a new website, language, or category is as simple as updating the config.json file, making the application scalable and easy to maintain.

Advantages

• Maintainability: Changes to the list of websites or their categories don't require code changes, reducing maintenance complexity.

User Experience: Provides a user-friendly interface where options are dynamically generated, offering a seamless and intuitive experience.

Demo

Conclusion

Our journey through scraping, cleaning, and deploying data from Moroccan news websites has been a testament to the power and flexibility of Python and its libraries. By leveraging tools like Selenium, Pandas, Streamlit, and Google Drive API, we've demonstrated a streamlined process that transforms raw data into accessible and actionable insights. This project not only showcases the technical capabilities of these tools but also highlights the potential for data-driven strategies in understanding and disseminating information effectively.

Acknowledgements

First and foremost, a heartfelt thanks to my dedicated teammates - Tajeddine Bourhim , Yahya NPC and @marwane khadrouf. Their expertise, creativity, and commitment were pivotal in turning this concept into reality. Their contributions in various aspects of the project, from data scraping to interface design, have been invaluable.

We also extend our gratitude to the founder of MDS and the initiator of the DataStart initiative Bahae Eddine Halim . This initiative not only kick-started our project but also inspired us to delve into the realm of data handling and analysis, contributing to a range of projects including this one.

this project stands as a collaborative effort, blending individual talents and shared vision. It's a celebration of teamwork, innovation, and the endless possibilities that open-source technology and data science bring to our world.

In today's globalized world, the ability to swiftly convert currencies is a necessity for travelers, businesses, and finance enthusiasts alike. In this blog, we delve into the fascinating realm of currency conversion, showcasing a Python-based web application that brings real-time exchange rates to your fingertips.

Join us on this journey as we unveil the inner workings of our currency converter, crafted with the powerful combination of Python, Streamlit, and BeautifulSoup. We'll explore the challenges and triumphs of sourcing current exchange rates, and the seamless user experience delivered by our intuitive web interface.

So sit back, relax, and prepare to embark on a voyage through the dynamic landscape of currency conversion, guided by the ingenuity of Python's versatile ecosystem. Let's dive in! 🌐💱

What is streamlit

Streamlit is a cutting-edge Python library that empowers developers to create interactive web applications with remarkable ease and speed. What sets Streamlit apart is its simplicity and focus on the developer experience. With just a few lines of Python code, developers can transform their data scripts into polished, user-friendly web apps. Streamlit handles all the heavy lifting behind the scenes, from data visualization to user input handling, allowing developers to focus on crafting engaging experiences for their users. Whether you're a seasoned developer or just starting out, Streamlit offers a seamless and intuitive platform for bringing your ideas to life on the web.

What is beautifulSoup

Beautiful Soup is a Python library renowned for its ability to parse HTML and XML documents, making web scraping and data extraction a breeze. It provides a powerful yet user-friendly interface for navigating and manipulating parsed documents, allowing developers to extract relevant information with ease. Beautiful Soup's intuitive syntax and robust functionality make it a go-to choice for extracting data from websites of all complexities. Whether you're scraping a simple webpage or traversing a complex hierarchy of elements, Beautiful Soup streamlines the process, enabling developers to focus on extracting insights from the data rather than wrestling with the intricacies of web parsing. With its versatility and reliability, Beautiful Soup continues to be a cornerstone tool for web scraping tasks across various domains and industries.

• Code snippet (main page)

import pandas as pd
import requests
from bs4 import BeautifulSoup
import streamlit as st


def rate_parser(input_curr, output_curr):
    url = f"https://www.xe.com/currencyconverter/convert/?Amount=1&From={input_curr}&To={output_curr}"
    content = requests.get(url).text
    soup = BeautifulSoup(content, 'html.parser')

    result_element = soup.find("p", class_="result__BigRate-sc-1bsijpp-1 dPdXSB")

    if result_element:
        currency_text = result_element.get_text().replace(',', '')  # Remove comma
        rate = float(currency_text.split()[0])
        return rate
    else:
        print(f"Element not found for {input_curr} to {output_curr}.")
        return None


def convert(base,dest,amount):
    rate = rate_parser(base,dest)
    new_amount = rate * amount
    return new_amount 


currencies = ["USD","EUR","CAD","MAD","GBP","AUD","JPY"]
#this is just a test list you can add any currency available at xe-currency

st.write("# Python based currency converter")

st.sidebar.write("### Currency converter:")

base = st.sidebar.selectbox("Enter a base currency:",currencies)

dest = st.sidebar.selectbox("Enter a destination currency:",currencies)

amount = st.sidebar.number_input("Enter an amount")

input = st.sidebar.button("Convert")

if input:
    current_rate = rate_parser(base,dest)
    output = convert(base,dest,amount)
    st.success("Success")
    st.write(f"## Current exchange rate between {base} and {dest} ")
    st.write(f"#### 1 {base} = ")
    st.write(f" ## :red[{current_rate}] {dest}")

    st.write("## Converted amount:")
    st.write(f" ### {amount} {base} = :red[{output}] {dest}")

Why we chose scraping over API

In our pursuit of building a Python-based currency converter web app, the decision to opt for web scraping over utilizing an API was rooted in the specific requirements of our project. While APIs offer a convenient way to access data, particularly for real-time exchange rates, we encountered limitations with free API plans, which typically impose restrictions on the number of requests allowed per day. This became a significant constraint when attempting to gather historical exchange rate data, as it necessitated a considerable number of requests.

In light of this, we shifted our approach to web scraping using BeautifulSoup. While scraping provided the flexibility needed for historical data retrieval, it introduced the trade-off of slower performance compared to APIs. The notable advantage, however, lay in the ability to extract comprehensive historical datasets without the constraints imposed by API rate limits. This strategic decision allowed us to balance our data acquisition requirements, ensuring the reliability and completeness of the information while acknowledging the potential trade-offs in speed associated with web scraping.

• Code snippet (historical data)

import pandas as pd
import requests
from datetime import timedelta, date
from bs4 import BeautifulSoup
import streamlit as st
import matplotlib.pyplot as plt
from matplotlib.ticker import MaxNLocator

#Create a function to return a list of each day between two dates

def daterange(start_date, end_date):
    list = []
    for n in range(int ((end_date - start_date).days)):
        list.append(start_date + timedelta(n))
    return list

#Create a function that scrapes historical data nb(all apis didn't work well)

def historical_data(start_date,end_date,base,dest):
    df = pd.DataFrame()
    times = daterange(start_date, end_date)
    for single_date in times:
        dfs = pd.read_html(f'https://www.xe.com/currencytables/?from={base}&date={single_date.strftime("%Y-%m-%d")}')[0]
        dfs['Date'] = single_date.strftime("%Y-%m-%d")
        df = pd.concat([df, dfs], ignore_index=True)

    df_curr=df.loc[df['Currency']==dest]
    df_curr = df_curr.reset_index(drop=True)
    df_curr.set_index('Date',inplace = True)
    return df_curr


#Create a list of all supported currencies

currencies = ["USD","EUR","CAD","MAD","GBP","AUD","JPY"]
#this is just a test list
#Expend the list as needed

#Create the UI using streamlit

st.write("# Historical Data:")
st.warning("This is originally a scraping webapp so choosing a large duration might cause substancial running time")
st.warning("Recommanded max duration : 1 Month")

base = st.sidebar.selectbox("Enter a base currency:",currencies)

dest = st.sidebar.selectbox("Enter a destination currency:",currencies)

start = st.sidebar.date_input("Enter start date:")

finish = st.sidebar.date_input("Enter finish date:")

input = st.sidebar.button("Confirm")

if input:
    if start == finish:
        st.error("Error: cannot process same date")
    with st.spinner('Wait for it...'):
        data = historical_data(start,finish,base,dest)
    st.success('Done!')
    st.table(data)
    st.write("## Plotting")

    st.write("### Static plot:")
    fig, ax = plt.subplots()
    ax.plot(data[f"{base} per unit"])
    ax.set_title(f'{base} to {dest} over Time')
    ax.xaxis.set_major_locator(MaxNLocator(nbins=6))
    st.pyplot(fig)

Example

Deployment

We chose Hugging Face for deployment due to its reputation as a leading platform for deploying machine learning models with ease and efficiency. Hugging Face offers a comprehensive suite of tools and services that streamline the deployment process, enabling us to seamlessly deploy our web app.

One of the key reasons for selecting Hugging Face is its user-friendly interface and robust documentation, which simplifies the deployment workflow and reduces the learning curve for developers.

Furthermore, Hugging Face's scalability and reliability were critical factors in our decision-making process. The platform's infrastructure is designed to handle high volumes of traffic and deliver consistent performance, making it well-suited for deploying production-ready applications.

Moreover, Hugging Face offers built-in features for model versioning, monitoring, and management, which facilitate seamless updates and maintenance of deployed models.

Overall, Hugging Face emerged as the optimal choice for deployment based on its reputation for reliability, scalability, ease of use, and comprehensive feature set, enabling us to deploy our machine learning models with confidence and efficiency.

https://huggingface.co/spaces/MoroccanDS/B7-Pyhton-based-currency-converter

Conclusion

In conclusion, embarking on this project to develop a Python-based currency converter web app has proven to be a highly rewarding endeavor, not only for the usefulness of the application but also for the invaluable skills gained throughout the process. By leveraging technologies such as Streamlit and BeautifulSoup, we've honed our abilities in web development, data extraction, and user interface design, equipping us with valuable tools for future projects.

• Introduction
• Data Scraping
• Data Cleaning
• Exploratory Data Analysis (EDA)
• Model Development
• Model Deployment
• Conclusion
• Acknowledgments

Introduction

Hello there! Are you up for an exciting journey through Morocco's real estate scene? From the bustling markets to the hidden gems of Marrakech and Chefchaouen, we're diving into the heart of it all. Join us as we decode the secrets of Moroccan property prices and unveil the magic of predictive analytics. Get ready to explore, analyze, and uncover the possibilities in this thrilling adventure! 🌟🔍

Data Scraping

In the realm of real estate analytics, data is king. But how do we access this treasure trove of information scattered across the vast expanse of the internet? Enter web scraping – the digital prospector's tool of choice.

What is Web Scraping?

Web scraping is the art of extracting data from websites, allowing us to collect valuable information for analysis and insights. In the context of our project, web scraping enables us to gather real estate listings, economic indicators, and other pertinent data from online sources.

Why is it Important?

In the dynamic world of real estate, having access to up-to-date and comprehensive data is crucial for making informed decisions. Web scraping empowers us to gather vast amounts of data efficiently, giving us a competitive edge in predicting real estate prices and understanding market trends.

Scraping Data with BeautifulSoup

Our journey begins with a prominent real estate website in Morocco, brimming with valuable listings and insights. Using the Python library BeautifulSoup, we embark on our quest to extract data from it with precision and finesse.

The Process:

Exploration: We start by inspecting the structure of the website, identifying the key elements containing the data we seek – from property listings to pricing information.

Scripting: Armed with our knowledge of HTML and CSS, we craft Python scripts leveraging BeautifulSoup to navigate through the website's code and extract the desired data.

Extraction: With surgical precision, our scripts traverse through the pages of the website, capturing essential details such as property names, prices, locations, area, and more.

Aggregation: As the data flows in, we gather and organize it into structured formats such as Excel spreadsheets, ready for further analysis and processing.

Python Code Snippets:

import requests
from bs4 import BeautifulSoup
import pandas as pd
from urllib.parse import urlparse, urljoin

# Define the base URL
baseurl = 'https://www.mu.ma/'

# Set to store unique product links
product_links = set()

# Loop through multiple pages of apartment listings
for page_num in range(1, 3):
    # Make a GET request to the page
    r = requests.get(f'https://www.mu.ma/fr/sc/appartements-a-vendre:p:{page_num}')
    soup = BeautifulSoup(r.content, 'html.parser')

   # Extract listings from the page
    listings = soup.find_all(class_='listingBox w100')

    # Loop through each listing and extract product links
    for listing in listings:
        links = listing.find_all('a', href=True)
        for link in links:
            product_link = link['href']

            # Check if the URL is valid
            if not urlparse(product_link).scheme:
                product_link = urljoin(baseurl, product_link)
                product_links.add(product_link)  # Add link to the set

# Initialize list to store scraped data
scraped_data = []

# Loop through each product link and scrape data
for link in product_links:
    try:
        # Make a GET request to the product page
        r = requests.get(link)
        soup = BeautifulSoup(r.content)
        # Extract relevant information from the page
        # (Code snippet continued from previous section...)
    except Exception as e:
        print('Error processing page:', link)
        print(e)
        continue

# Convert scraped data to DataFrame and save to Excel
df = pd.DataFrame(scraped_data)
df.to_excel('scraped_data.xlsx', index=False)

With web scraping as our trusty pickaxe, we delve into the digital mines of the website, extracting nuggets of real estate data to fuel our predictive models and illuminate the path to informed decision-making. Let the data adventure begin! 🏠💻

Data Cleaning

Why Data Cleaning Matters:

Before we can uncover the hidden patterns and insights within our data, we must first ensure its integrity and quality. Data cleaning plays a pivotal role in this process, serving as the foundation upon which our analyses and models are built. By removing inconsistencies, handling missing values, and standardizing formats, we pave the way for accurate and reliable results.

Steps in Data Cleaning:

Removing Duplicates: Duplicate entries can skew our analyses and lead to erroneous conclusions. By identifying and eliminating duplicates, we ensure that each observation contributes meaningfully to our insights.

Handling Missing Values: Missing data is a common challenge in real-world datasets. Whether due to human error or system limitations, missing values must be addressed to maintain the integrity of our analyses. Strategies such as imputation or removal can be employed based on the nature and context of the missing data.

Standardizing Data Types: Inconsistent data types can impede our ability to perform meaningful calculations and comparisons. Standardizing data types ensures uniformity and facilitates seamless data manipulation and analysis.

Python Code Snippets:

# Load the scraped data after some cleaning with VBA Macros

df = pd.read_excel('scraped_data.xlsx')

# Remove duplicate rows

df.drop_duplicates(inplace=True)

# Drop rows with missing prices

df.dropna(subset=['price'], inplace=True)

# Fill missing values in 'city' with corresponding values from 'secteur'

df['city'].fillna(df['secteur'], inplace=True)

# Fill missing values in 'sdb' and 'chambres' with the mean

mean_sdb = df['sdb'].mean()

mean_chambres = df['chambres'].mean()

mean_surface = df['surface'].mean()

df['sdb'].fillna(mean_sdb, inplace=True)

df['chambres'].fillna(mean_chambres, inplace=True)

df['surface'].fillna(mean_surface, inplace=True)

# Fill missing values in 'pieces' with the sum of 'sdb' and 'chambres'

df['pieces'].fillna(df['sdb'] + df['chambres'], inplace=True)

# One-hot encode 'secteur' and 'city' columns

df = pd.get_dummies(df, columns=['secteur', 'city'])

# Display the cleaned DataFrame

print(df.head())

Example:

Let's say our original dataset contains the following entries:

After applying data cleaning techniques, our cleaned dataset might look like this:

Conclusion:

Data cleaning is the cornerstone of robust data analysis. By ensuring the cleanliness and consistency of our datasets, we lay the groundwork for accurate insights and informed decision-making. With our data now polished to perfection, we are ready to embark on the next stage of our real estate journey

Exploratory Data Analysis (EDA)

Understanding the Dataset:

Exploratory Data Analysis (EDA) serves as our compass, guiding us through the vast landscape of real estate data. It empowers us to uncover hidden patterns, identify outliers, and gain valuable insights into the dynamics of the market.

Significance of EDA:

Understanding Real Estate Dynamics: EDA allows us to delve deep into the intricacies of the real estate dataset, unraveling the relationships between various factors such as property characteristics, location, area, and pricing.

Identifying Patterns and Trends: By analyzing descriptive statistics and visualizations, we can identify trends over time, seasonal fluctuations, and spatial disparities in property prices.

Informing Decision-Making: Insights gleaned from EDA serve as the cornerstone for informed decision-making, whether it be for investors, developers, or policymakers.

Visualizations and Descriptive Statistics:

Distribution of Real Estate Prices: Histograms and boxplots provide a snapshot of the distribution of property prices, highlighting central tendencies, variability, and potential outliers.

Relationships Between Variables: Scatter plots and correlation matrices help us explore the relationships between different variables, such as property size, number of rooms, and prices, shedding light on potential predictors of property value.

Temporal Trends: Time series plots allow us to visualize temporal trends in property prices, discerning patterns, seasonality, and long-term trends.

Geospatial Visualizations:

Interactive Maps: Utilizing geospatial data, we can create interactive maps to visualize property locations, hotspots, and regional disparities in prices. This allows stakeholders to explore the real estate landscape at a glance and identify areas of interest.

Heatmaps: Heatmaps offer a bird's-eye view of property density and price distribution, providing valuable insights into market saturation and demand hotspots.

Conclusion:

Exploratory Data Analysis serves as our compass, guiding us through the labyrinth of real estate data. By unraveling patterns, trends, and spatial dynamics, EDA equips us with the insights needed to navigate the complexities of the real estate market with confidence and clarity. With our eyes opened to the rich tapestry of real estate data, we are poised to unlock its full potential and drive informed decision-making in the ever-evolving landscape of real estate. 📊🏠

Model Development

Model development marks the culmination of our journey, where we harness the power of machine learning to predict real estate prices with precision and accuracy. This transformative process involves a series of meticulous steps, each contributing to the creation of robust predictive models.

Model Development Process:

Data Normalization: We embark on our journey by normalizing the data, ensuring that all features are on a consistent scale. This step prevents certain features from dominating the model training process and ensures optimal performance.

Feature Selection: With a myriad of features at our disposal, we carefully select the most influential ones to include in our predictive models. Through feature selection techniques, we prioritize variables that exhibit strong correlations with property prices and contribute meaningfully to the predictive power of our models.

Model Training: Armed with a curated dataset and selected features, we embark on the model training phase. Leveraging powerful machine learning libraries such as TensorFlow, we train regression models to learn from historical data and discern intricate patterns in real estate pricing dynamics.

Evaluation Metrics: As stewards of data-driven decision-making, we rely on rigorous evaluation metrics to assess the performance of our models. Metrics such as Mean Squared Error (MSE) and Mean Absolute Error (MAE) serve as our compass, guiding us towards models that exhibit optimal predictive accuracy and generalization.

Harnessing TensorFlow and Beyond:

TensorFlow: TensorFlow stands as our stalwart companion on the journey of model development, providing a versatile framework for building and training regression models. With its intuitive interface and powerful capabilities, TensorFlow empowers us to bring our predictive visions to life with elegance and efficiency.

Machine Learning Libraries: In addition to TensorFlow, we harness a diverse array of machine learning libraries such as scikit-learn and Keras to augment our model development efforts. These libraries offer a rich ecosystem of algorithms and tools, enabling us to experiment, iterate, and refine our predictive models with finesse.

python Code Snippet

from sklearn.model_selection import train_test_split

from sklearn.preprocessing import StandardScaler

from sklearn.metrics import mean_squared_error, mean_absolute_error

# Load the dataset and preprocess features

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

scaler = StandardScaler()

X_train_scaled = scaler.fit_transform(X_train)

X_test_scaled = scaler.transform(X_test)

# Define and train the TensorFlow regression model

model = tf.keras.Sequential([

tf.keras.layers.Dense(64, activation='relu', input_shape=(X_train_scaled.shape[1],)),

tf.keras.layers.Dense(64, activation='relu'),

tf.keras.layers.Dense(1)

])

model.compile(optimizer='adam', loss='mean_squared_error')

model.fit(X_train_scaled, y_train, epochs=100, batch_size=32, verbose=1)

# Evaluate the model

y_pred = model.predict(X_test_scaled)

mse = mean_squared_error(y_test, y_pred)

mae = mean_absolute_error(y_test, y_pred)

print(f'Mean Squared Error: {mse}')

print(f'Mean Absolute Error: {mae}')

Conclusion:

Model development serves as the cornerstone of our predictive journey, where we transform data into actionable insights and empower stakeholders to make informed decisions in the dynamic realm of real estate. Through meticulous attention to detail, rigorous experimentation, and the judicious application of machine learning techniques, we pave the way for predictive models that illuminate the path forward with clarity and confidence. 🤖🏠📈

Model Deployment

Deployment Process:

As we approach the culmination of our journey, it's time to unleash the predictive prowess of our meticulously crafted models into the real world. Model deployment represents the pivotal moment when theoretical concepts seamlessly transition into practical applications, offering actionable insights and informed decision-making capabilities to stakeholders.

Deployment Landscape:

Real-World Integration: Our trained models are seamlessly integrated into real-world environments, where they stand ready to analyze incoming data and provide valuable predictions on real estate prices. Whether it's assisting homebuyers in making informed purchase decisions or aiding industry professionals in strategic planning, our deployed models serve as beacons of predictive wisdom.

Deployment Overview: Our deployment offers a glimpse into the intuitive interface through which users can interact with the deployed model. By inputting relevant property features such as surface area, number of rooms, and location, users can harness the predictive power of our models to obtain accurate price predictions in a matter of seconds.

User Interaction: Interacting with our deployed model is as simple as entering the desired property features into the designated input fields. With just a few clicks, users gain access to personalized price predictions tailored to their specific requirements, empowering them to navigate the intricacies of the real estate market with confidence and clarity.

Ready to experience the magic of predictive analytics firsthand? Explore our deployment and witness the transformative potential of data-driven insights in action. Click here to embark on a journey of discovery and unlock the secrets of real estate pricing dynamics with just a click of a button.

https://huggingface.co/spaces/saaara/real_estate_price_prediction

Conclusion:

Model deployment represents the culmination of our journey, where theoretical concepts are transformed into tangible solutions that empower individuals and organizations to make informed decisions in the ever-evolving landscape of real estate. Through seamless integration, intuitive interfaces, and the democratization of predictive analytics, we pave the way for a future where data-driven insights drive meaningful change and innovation.

Conclusion

We've made our data analysis and modeling process interactive and accessible by sharing our Google Colab notebook. Dive deeper into the intricacies of our real estate analysis, run code cells, visualize data, and even experiment with our predictive models here:

https://colab.research.google.com/drive/1sWd5QhPXL0MpLsRsYBb7JuxK8uDTYCaq?authuser=1#scrollTo=iJF6DFW618jR

As our journey through the labyrinth of Moroccan real estate draws to a close, we stand amidst a landscape adorned with insights, revelations, and transformative discoveries. Through the collective efforts of our dedicated team, we have unearthed invaluable treasures that illuminate the intricate dynamics of real estate pricing in Morocco. Our models exhibit remarkable accuracy in forecasting real estate prices, leveraging a nuanced understanding of location, amenities, and economic trends. By harnessing the power of web scraping and exploratory data analysis, we have curated a comprehensive dataset, laying the foundation for informed decision-making. With the deployment of our models, we bridge the gap between theory and practice, offering intuitive interfaces to access predictive analytics, empowering users to navigate the complexities of the real estate market with confidence. Looking ahead, we remain committed to continuous innovation, exploring novel methodologies and technologies to enhance the accuracy of our models. While reflecting on our journey, we recognize areas for improvement, acknowledging challenges faced and lessons learned.

A Call to Action:

Embark on your own journey of exploration and discovery within the realm of Moroccan real estate. Whether you're a seasoned professional, an aspiring enthusiast, or a curious explorer, there's a wealth of insights and opportunities waiting to be uncovered. Connect with our team to learn more about our endeavors, collaborate on future projects, or simply indulge in the fascinating world of real estate analytics. Together, let's chart a course towards a future where data-driven insights illuminate the path to prosperity and growth. 🌟🔍🏠

Acknowledgments

I would like to express my sincere appreciation to my dedicated team members – Sara M'HAMDI, Imane KARAM, and Asmae EL-GHEZZAZ – whose expertise and commitment have been invaluable throughout this project. Your hard work, collaboration, and enthusiasm have truly made a difference. As the team leader, I am incredibly proud to have worked alongside such talented individuals.

I also want to acknowledge Bahae Eddine Halim, the founder of the Moroccan Data Science (MDS) community, whose initiative, DataStart First Edition, provided the platform for our project. His dedication to fostering a supportive environment for data enthusiasts in Morocco has been instrumental in our journey. Lastly, we thank the broader data science community for their support and encouragement. Your enthusiasm and engagement have motivated us to push boundaries and continuously strive for excellence in our endeavors.

To all those who have contributed to this project – mentors, team members, and supporters – we express our heartfelt thanks. Your collective efforts have been integral to the success of this endeavor, and we look forward to continued collaboration and growth in the future.

In the heart of Morocco, a tech revolution is brewing, and at the forefront stands the Moroccan Data Scientists (MDS) community. MDS is not just a hub for technological advancement; it's a philosophy that nurtures inquisitive minds and empowers aspiring data scientists. Let's delve into the unique initiatives and vibrant community that make MDS a beacon of innovation in the Moroccan tech landscape.

MDS Philosophy

At MDS, learning is not just about academic excellence; it's about fostering confident, creative thinkers. The community is dedicated to merging social-emotional development with technical proficiency, ensuring that its members not only excel in data science but also emerge as impactful leaders in their industries.

Discover More About MDS

MDS takes pride in its active members, qualified professionals, and a plethora of activities and collaborative projects. Founder Bahae Eddine Halim envisions inspiring members to dream big and become trailblazers in the realms of technology and data science.

Explore our Specialties

The learning experience at MDS is diverse and comprehensive. From sophisticated data analysis techniques and machine learning to artificial intelligence engineering and data visualization, members are equipped with skills that transcend traditional boundaries.

Our "Innovative" Initiatives at MDS

MDS goes beyond data understanding; it embeds it in the rich tapestry of Moroccan culture. Initiatives like the #DataStart program kickstart data science projects addressing local challenges. Regular webinars featuring experts in data science and AI keep the community abreast of the latest trends. Data-driven publications by members showcase how data science is shaping various sectors in Morocco.

Conclusion

As we navigate through the virtual corridors of Moroccan Data Scientists, it's evident that MDS is not just a community; it's a movement. A movement that empowers, inspires, and connects data enthusiasts in Morocco. With its unique blend of technology and cultural relevance, MDS is shaping the future of tech in Morocco, one data scientist at a time.