Course No 2 Using Python to Interact with the Operating System Rough Notes

MAIN COURSE Google IT Automation with Python Professional Certificate

Having 6 crouse

1st course was Python Crash which I have DONE got certificate. My friend say 'bati bana kar gand meen dalo'

Course No 2 Using Python to Interact with the Operating System

------------------------------

course 2 week 5 added 

By inputting emp name the program will return its email address
By inputing   Blossom Gill program will return his email aress blossom@abc.edu

Our job is to 
1. add a test means  test cases like basic_test Edge_test see the bugs, 
2. make the necessary corrections in  email.py Python file to correct the bugs  
3. verify that all the tests pass to make sure the script works! 
4.  This is The End Lab . Best of luck! and Congrautulations

Once again, in other words, same matter/problem as mentioned above
What you'll do

In this lab, you will:

  • Write a simple test (basic test )to check for basic functionality
  • Write a test to check for edge cases by inputting only 1. First name or last name 2.  first, last and middle name
  • Correct code with a try/except statement
===============initial file emails.py given After checking with differferent para paramter/augument we found many errorors like 
1. email.py with no augument
2. email.py with one name first or last name only
3. Given 3/4 augument
So step by step first we make testcase emails_test.py file after getting bugs throught e test file we modify emails.py accordingly
initial file emails.py is as undder
user-emails.csv file used which is mentioned below
=====================================================================
#!/usr/bin/env python3
import unittest
from emails import find_email
class TestFile(unittest.TestCase):
  def test_basic(self):
    testcase = [None, "Bree", "Campbell"]
    expected = "breee@abc.edu"
    self.assertEqual(find_email(testcase), expected)
  def test_one_name(self):
    testcase = [None, "John"]
    expected = "Missing parameters"
    self.assertEqual(find_email(testcase), expected)
if __name__ == '__main__':
  unittest.main()  
=============================================================================
~/Week-6$ cat user-emails.csv 
Blossom Gill,blossom@abc.edu,
Hayes Delgado,nonummy@abc.edu
Petra Jones,ac@abc.edu
Oleg Noel,noel@abc.edu
Ahmed Miller,ahmed.miller@abc.edu
Macaulay Douglas,mdouglas@abc.edu
Aurora Grant,enim.non@abc.edu
Madison Mcintosh,mcintosh@abc.edu
Montana Powell,montanap@abc.edu
Rogan Robinson,rr.robinson@abc.edu
Simon Rivera,sri@abc.edu
Benedict Pacheco,bpacheco@abc.edu
Maisie Hendrix,mai.hendrix@abc.edu
Xaviera Gould,xlg@abc.edu
Oren Rollins,oren@abc.edu
Flavia Santiago,flavia@abc.edu
Jackson Owens,jacksonowens@abc.edu
Britanni Humphrey,britanni@abc.edu
Kirk Nixon,kirknixon@abc.edu
Bree Campbell,breee@abc.edu~/Week-6$
===============================
-----------------------nano emails_test.py---------------------------------------------------
#!/usr/bin/env python3
import unittest
from emails import find_email
class EmailsTest(unittest.TestCase):
  def test_basic(self):
    testcase = [None, "Bree", "Campbell"]
    expected = "breee@abc.edu"
    self.assertEqual(find_email(testcase), expected)
  def test_one_name(self):
    testcase = [None, "John"]
    expected = "Missing parameters"
    self.assertEqual(find_email(testcase), expected)
  def test_two_name(self):
    testcase = [None, "Roy","Cooper"]
    expected = "No email address found"
    self.assertEqual(find_email(testcase), expected)
if __name__ == '__main__':
  unittest.main()



-----------final version----emails.py file================
#!/usr/bin/env python3
import csv
import sys
def populate_dictionary(filename):
  """Populate a dictionary with name/email pairs for easy lookup."""
  email_dict = {}
  with open(filename) as csvfile:
    lines = csv.reader(csvfile, delimiter = ',')
    for row in lines:
      name = str(row[0].lower())
      email_dict[name] = row[1]
  return email_dict
def find_email(argv):
  """ Return an email address based on the username given."""
  # Create the username based on the command line input.
  try:
    fullname = str(argv[1] + " " + argv[2])
    # Preprocess the data
    email_dict = populate_dictionary('/home/{{ username }}/data/user_emails.csv')
     # If email exists, print it
    if email_dict.get(fullname.lower()):
      return email_dict.get(fullname.lower())
    else:
      return "No email address found"
  except IndexError:
    return "Missing parameters"
def main():
  print(find_email(sys.argv))
if __name__ == "__main__":
  main()
-----------------------use_emails.csv-----------fuke
Blossom Gill,blossom@abc.edu,
Hayes Delgado,nonummy@abc.edu
Petra Jones,ac@abc.edu
Oleg Noel,noel@abc.edu
Ahmed Miller,ahmed.miller@abc.edu
Macaulay Douglas,mdouglas@abc.edu
Aurora Grant,enim.non@abc.edu
Madison Mcintosh,mcintosh@abc.edu
Montana Powell,montanap@abc.edu
Rogan Robinson,rr.robinson@abc.edu
Simon Rivera,sri@abc.edu
Benedict Pacheco,bpacheco@abc.edu
Maisie Hendrix,mai.hendrix@abc.edu
Xaviera Gould,xlg@abc.edu
Oren Rollins,oren@abc.edu
Flavia Santiago,flavia@abc.edu
Jackson Owens,jacksonowens@abc.edu
Britanni Humphrey,britanni@abc.edu
Kirk Nixon,kirknixon@abc.edu
Bree Campbell,breee@abc.edu
Getting Ready for the Final Project

1.understanding Problem defination

2. Research

3. Planning

 SU abssite /var/adm => cat syslog | cut -d' ' -f1| uniq -c

 804 Jun
 324 Jul
 265 Aug
 109 Sep
  74 Oct
 258 Nov
 134 Dec
  98 Jan
  54 Feb
  52 Mar
  52 Apr
 121 May
 130 Jun
 152 Jul
 446 Aug
 159 Sep

-----------------------------

All Python Programs .py and Bash Script available at one place
-----------------------------------------------------------------------

  • Bash Scripting (Linux) Resources

    Check out the following links for more information:

    • https://ryanstutorials.net/bash-scripting-tutorial/

    • https://linuxconfig.org/bash-scripting-tutorial-for-beginners


    • ------------------------------------------------

      -----------------------------------------------------

    • ~/Week-6$ cat fruits.sh for fruit in orange mango banana tomatoooo do echo I like $fruit done ~/Week-6$ ./fruits.sh I like orange I like mango I like banana I like tomatoooo ~/Week-6$


    • ~/Week-6$ cat random-exit.py
      #!/usr/bin/env python3
      import sys
      import random
      value=random.randint(0,5)
      print('exit=',(value))
      sys.exit(value)
    • ---------------------------------
    • ~/Week-6$ cat trycmd.sh n=0 command=$1 while ! $command && [ $n -le 5 ] #while ! $command ; do ((n+=1)) #sleep $n echo Tried:$n done ~/Week-6$
    • -----

    -------------------------------

    • -----------------------------------------


        • ~/Python$ cat loop.sh n=1 while [ $n -le 5 ]; do echo $n ((n+=1)) done
        • ----------------------------



  • cut -dseparator -ffields file: for each line in the given file, splits the line according to the given separator and prints the given fields (starting from 1)

------------------------------------------------------



Correct

Way to go! This is a practical application of using Python (and some extra hardware, in this case) to automate a task, freeing up a human's time. The solutions can be complex if the return in saved human time warrants it.

-------------------------------------------

CHAPTER2 AUTOMATION::USING PYTHON TO INTERACT WITH OS WEEK1


What is the purpose of using a shebang line in a Python file?

TERM
IMAGE

To specify beforehand what command to use when running the script (Inserting a shebang line (such as #!/usr/bin/env python3) as the first line tells the operating system what command we want to use to execute the script.)

DEFINITION

Proper code reuse is important, so let's see if you were paying attention. Which of the following lines will allow you to use the datetime module in your script?

TERM
IMAGE

import datetime. (We use the import command to import any module located in the PATH directory. We can also use import as to assign a localName variable to the imported module.)

DEFINITION

Which of the following is NOT an example of code completion?

TERM
IMAGE

After you type the letters "ret", your IDE finishes your sentence with return. (YES)


After typing the opening parentheses after a function, your IDE inputs the final parentheses. (YES)


After typing "def", your IDE detects that you are typing a function and highlights it. (NO)

(Syntax highlighting detects the language we're writing our code in and highlights the pieces of code that make up the syntax of the language.)


After typing "def", your IDE detects that you are typing a function and highlights it. (YES)


After typing the first few letters of an existing variable, your IDE suggests that variable, highlighting the suggestion. (YES)

DEFINITION

Which of the following is the most modern, up-to-date version of Python?

TERM
IMAGE

Python 3 is the latest version of Python, with Python 3.8.0 being released on October 14, 2019.

DEFINITION

Which of the following operating systems is compatible with Python 3?

TERM
IMAGE

Redhat Linux

Microsoft Windows

Apple MacOS

(Python is a cross-platform language. You can use it on Windows, macOS, Linux, and even on lesser-known Unix variants like FreeBSD.)

DEFINITION

Which of the following operating systems does not run on a Linux kernel?

TERM
IMAGE

MacOS (Mac OS is a proprietary operating system designed by Apple and uses a proprietary kernel based on BSD.)

DEFINITION

f we want to check to see what version of Python is installed, what would we type into the command line? Select all that apply.

TERM
IMAGE

python -V (Typing python -V (note the capital V) at the command line will tell you if Python is currently installed and if so, what version.)


python --version (Typing python --version (note the double dashes) at the command line will tell you if Python is currently installed and if so, what version.)

DEFINITION

When your IDE automatically creates an indent for you, this is known as what?

TERM
IMAGE

code completion (Code completion is an IDE feature that takes educated guesses about what you might be trying to type next, and offers suggestions to complete it for you.)

DEFINITION

Can you identify the error in the following code?


#!/usr/bin/env python3

import numpy as np

def numpyArray():

x = np.array([[1, 2, 3], [4, 5, 6]], np.int32)

y = numpy.array([[3, 6, 2], [9, 12, 8]], np.int32)

return x*y

print(numpyArray())

TERM
IMAGE

The y variable is not calling the numpy module properly (While the x variable is calling numpy using its declared local name, y is not using the local name. This will result in an error.)

DEFINITION

Which type of programming language is read and converted to machine code before runtime, allowing for more efficient code?

TERM
IMAGE

Compiled Language (A compiled language is translated into code readable by the target machine during development using a compiler.)

DEFINITION

Which of the following is NOT an IDE or code editor?

TERM
IMAGE

Eclipse (YES)

Atom(YES)

PyCharm(YES)

pip (NO) -- The package manager pip is used in Python to install packages from repositories such as PyPI.

DEFINITION

What does the PATH variable do?

TERM
IMAGE

tells the operating system where to find executables

DEFINITION

In terms of automation, which of the following is an example of scalability?

TERM
IMAGE

An IT engineer writes a script to compile a report on each machine's uptime and downtime for the day and email it to relevant parties every evening.

(Scalability means that when more work is added to a system, the system can do whatever it needs to complete the work.)

DEFINITION

Which of the following correctly describes the Pareto Principle?

TERM
IMAGE

One fifth of the sysadmin tasks you perform comprise four fifths of your total workload.

(The Pareto Principle states that 20% of the system administration tasks you perform are responsible for 80% of your workload. Therefore, identifying and automating those tasks will put your productivity through the roof!)

DEFINITION

Which of the following could be a good example of when to use an automation script?

TERM
IMAGE

Detect dangerously high CPU usage levels across a network and scale back the CPU clock speeds of those devices, or shut them down to prevent overheating (These days most consumer devices do this on their own, but in the case of custom hardware or other specific use cases such as cluster networks, automation would fit this task nicely and reduce failover.)

DEFINITION

At a manufacturing plant, an employee spends several minutes each hour noting uptime and downtime for each of the machines they are running. Which of the following ideas would best automate this process?

TERM
IMAGE

Add the analog Internet of Things (IoT) module to each machine, in order to detect their power states and write a script to records uptime and downtime, reporting hourly (This is a practical application of using Python (and some extra hardware, in this case) to automate a task, freeing up a human's time. The solutions can be complex if the return in saved human time warrants it.)

DEFINITION

One important aspect of automation is forensic value. Which of the following statements describes this term correctly?

TERM
IMAGE

It is important to automated processes to leave extensive logs so when errors occur , they can be properly investigated (Forensic value, in relation to automation, is the value we get while debugging from properly logging every action our automation scripts take.)

DEFINITION

An employee at a technical support company is required to collate reports into a single file and send that file via email three times a day, five days a week for one month, on top of his other duties. It takes him about 15 minutes each time. He has discovered a way to automate the process, but it will take him at least 10 hours to code the automation script. Which of the following equations will help them decide whether it's worth automating the process?

TERM
IMAGE

if [10 hours to automate < (15minutes * 60 times/month)] then automate (With 10 hours to automate, the employee will start saving time before the month is over.)

DEFINITION

A company is looking at automating one of their internal processes and wants to determine if automating a process would save labor time this year. The company uses the formula [time_to_automate < (time_to_perform * amount_of_times_done) to decide whether automation is worthwhile. The process normally takes about 10 minutes every week. The automation process itself will take 40 hours total to complete. Using the formula, how many weeks will it be before the company starts saving time on the process?

TERM
IMAGE

240 weeks (It's safe to say that the company won't find it worth it's time to automate.)

DEFINITION

Which of the following are valid methods to prevent silent automation errors? (Check all that apply)

TERM
IMAGE

Email notification about errors

(Email notifications for errors or task completions can help keep track of automated processes.)


Internal Issue Tracker Entries

(Internal issue tracker entries are created as part of reporting on errors in our automation script in this lesson.)


regular consistency check

(Automated consistency checks, such as hash checks on backups, can help identify problems ahead of time.)

DEFINITION


TERM
IMAGE


DEFINITION


--------------------------

import shutil
ds=shutil.disk_usage('/')
 df
usage(total=1760680976384, used=44638605312, free=1626529214464)
 df.total,df.used,df.free
(1760680976384, 44638605312, 1626529214464)
 df.free*2
3253058428928
import psutil
psutil.cpu_percent(.1)

==================
To change Terminal window green Font $ echo -e "\033[32m"

Practice Notebook - Errors and Exceptions

https://github.com/tamerlan2210/Using-Python-to-Interact-with-the-Operating-System/blob/main/C2M5L4_Errors_and_Exceptions-V2.ipynb

--------------------------------------------------------

When a try block is not able to execute a function, which of the following return examples will an exception block most likely NOT return?

Correct

Awesome! An exception is not meant to produce an error, but to bypass it.

-----------------------------------------------------------------------------------

Test the software ability and behavious under strees conditions

Correct

Way to go! Load testing verifies the behavior of the software remains consistent under conditions of significant load.

----------------------------------------------------------------

_______A test written after bug is identified 

Correct

Excellent! Regression testing is a type of software test used to confirm that a recent program or code change has not adversely affected existing features, by re-executing a full or partial selection test cases.

Verifying an automation script works well in the overall system and external entities which type test.

Correct

Great job! This test verifies that the different parts of the overall system interact as expected.

---------------------------------------------------------------

Practice Notebook - Unit Tests and Edge Cases

Below we have some code that makes a list of specific letters found in any string. If you run it, you can see what it does.

In [1]:
['a', 'b']

Fill in the blanks below to create an automatic unit test that verifies whether input strings have the correct list of string matches.
In [2]:
Now that your automatic test is coded, you need to call the unittest.main( ) function to run the test. It is important to note that the configuration for running unit tests in Jupyter is different than running unit tests from the command line. Running unittest.main( ) in Jupyter will result in an error. You can see this by runnig the following cell to execute your automatic test.
In [3]:
E
======================================================================
ERROR: /home/jovyan/ (unittest.loader._FailedTest)
----------------------------------------------------------------------
AttributeError: module '__main__' has no attribute '/home/jovyan/'

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (errors=1)

An exception has occurred, use %tb to see the full traceback.

SystemExit: True



Yikes! SystemExit: True means an error occurred, as expected. The reason is that unittest.main( ) looks at sys.argv. In Jupyter, by default, the first parameter of sys.argv is what started the Jupyter kernel which is not the case when executing it from the command line. This default parameter is passed into unittest.main( ) as an attribute when you don't explicitly pass it attributes and is therefore what causes the error about the kernel connection file not being a valid attribute. Passing an explicit list to unittest.main( ) prevents it from looking at sys.argv.

Let's pass it the list ['first-arg-is-ignored'] for example. In addition, we will pass it the parameter exit = False to prevent unittest.main( ) from shutting down the kernel process. Run the following cell with the argv and exit parameters passed into unittest.main( ) to rerun your automatic test.
In [ ]:
Did your automatic test pass? Was OK the result? If not, go back to your automatic test code and make sure you filled in the blanks correctly. If your automatic test passed, great! You have successfully filled in the gaps to create an automatic test that verifies whether input strings have the correct list of string matches.

------------------------------------------

Manual Testing and Automated Testing

Unit Test: is to test only a small isolated part of the software that either it is working correctly or not. Getting required Results or not. In unit test programs functions and methods are tested. Unit test cant not test/evaluates external errors/factors lime networking issue , full disk space or light failure or disk failures etc.
We must not run any test on production environment. We must test on a separate space other than production 

What is software testing:
    The process of evaluating codes to determine wheater or not it does what expect you
    it do.
    In other it is evaluating codes either the software is producing the required results or     not.  To find the errors and defects of coding/software.

Types of testing:
1. Manual 
2. Automated
3. Unit test
4. Integrated test
5. Test driven developement

Testing by changing different parametres is MANUAL testing.
Automated testing you have to write testing codes to test the software.
Automatic tests will always get the same expected result if the software code is good.

Test cases automatically test with a range of possible values to verify the program’s behavior.

Codifying tests into its own software and code that can be run to verify that our programs do what we expect them to do is automatic testing.
For automatic testing we have to write code to test the program. Using different test cases and different parameters (as many different values as possible) to check the output results are according to Required/Expected result.

The most basic way of testing a script is to use different parameters and get the expected results.

Edge Test:
        by inputting extreme ends of ranges like empty_test to longest string to like 'Saeed Hassan Hisbani Balouch S/O Muhammad Hassan Hisbani' or int values input from -9999999999999999999 to +99999999999999999999999999999999
--------------------------------------------------------

What module can you load to use a bunch of testing methods for your unit tests?

Correct

Nice job! This module provides a TestCase class with a bunch of testing methods.

-------------------------------------------------------

The advantage of running automated tests is that they will always get the same expected ___ if the software code is good.

Correct

Way to go! Automatic tests will always get the same expected result if the software code is good.

-----------------------------------------
Following Programs used in Testing

----this is rearrange.py
#!/usr/bin/env python3
import re
def rearrange_name(name):
    r=re.search(r'(^[\w .]*) ([\w .]*$)',name)
    return '{} {}'.format(r[2],r[1])
--------This is rearrange_test.py--------------------------------
#!/usr/bin/env python3
from rearrange import rearrange_name
import unittest
class TestRearrange(unittest.TestCase):
  def test_basic(self):   # test_basic is hard coded other wise I can not get results when I worte basic_test(self):
    testcase = 'Lovelace Ada'
    expected = 'Ada Lovelace'
    self.assertEqual(rearrange_name(testcase),expected)

 def test_empty(self):
    testcase=''
    expected=''
    self.assertEqual(rearrange_name(testcase),expected)

unittest.main()
----------------------------------------

What module can you load to use a bunch of testing methods for your unit tests?

Correct

Nice job! This module provides a TestCase class with a bunch of testing methods.

-----------------------------syslog -----------------------=

#!/usr/bin/env python3

import sys

import os

import re

def error_search(log_file):

  error = input("What is the error? ")

  returned_errors = []

  with open(log_file, mode='r',encoding='UTF-8') as file:

    for log in  file.readlines():

      error_patterns = ["error"]

      for i in range(len(error.split(' '))):

        error_patterns.append(r"{}".format(error.split(' ')[i].lower()))

      if all(re.search(error_pattern, log.lower()) for error_pattern in error_patterns):

        returned_errors.append(log)

    file.close()

  return returned_errors

  

def file_output(returned_errors):

  with open(os.path.expanduser('~') + '/data/errors_found.log', 'w') as file:

    for error in returned_errors:

      file.write(error)

    file.close()

if __name__ == "__main__":

  log_file = sys.argv[1]

  returned_errors = error_search(log_file)

  file_output(returned_errors)

  sys.exit(0)

=========

--------------------------------------------------------

July 31 22:05:31 mycomputername process[37921]: ERROR Process failed

July 31 22:07:54 mycomputername cacheclient[12017]: ERROR Process failed

July 31 22:14:37 mycomputername utility[78832]: INFO Healthy resource usage

July 31 22:22:34 mycomputername updater[78750]: ERROR ID: 10t

July 31 22:41:00 mycomputername jam_tag=psim[40956]: ERROR The cake is a lie!

July 31 22:42:51 mycomputername NetworkManager[28895]: INFO I'm sorry Dave. I'm afraid I can't do that

July 31 23:15:22 mycomputername jam_tag=psim[75013]: ERROR Failed process [13966]

July 31 23:17:05 mycomputername system[88316]: ERROR 418: I'm a teapot

July 31 23:20:09 mycomputername system[42918]: ERROR Out of ink

July 31 23:28:49 mycomputername utility[40452]: INFO Checking process [16121]

July 31 23:29:25 mycomputername utility[22941]: WARN Failed to start CPU thread[39016]

July 31 23:30:46 mycomputername process[99294]: WARN Computer needs to be turned off and on again

July 31 23:35:49 mycomputername dhcpclient[10726]: INFO Googling the answer

July 31 23:36:29 mycomputername updater[86053]: ERROR AssertionError 'False' is not 'True'

July 31 23:40:05 mycomputername utility[88068]: INFO Generating Logs

July 31 23:48:17 mycomputername CRON[28813]: WARN Please send help I am stuck inside the internet

July 31 23:53:58 mycomputername CRON[59985]: INFO Successfully connected


==================syslog data file===================================sysl

 This is hand medfd syslog: USER (saeed)
no usr in 2nd line 
12345678910111213141516171819
Jul 6 14:01:23 computer.name CRON[29440]: USER (good_user)
Jul 6 14:02:08 computer.name jam_tag=psim[29187]: (UUID:006)
Jul 6 14:03:01 computer.name CRON[29440]: USER (naughty_user)
Jul 6 14:03:40 computer.name cacheclient[29807]: start syncing from \
Jul 6 14:04:01 computer.name CRON[29440]: USER (naughty_user)
Jul 6 14:05:01 computer.name CRON[29440]: USER (naughty_user)

-------------------------------------------------

#!/usr/bin/env python3
import re
import sys
logfile = sys.argv[1]
print(sys.argv[0])
names={}
with open(logfile) as f:
    for line in f.readlines():
        if 'USER' not in line:
            continue
  #Jul 6 14:01:23 computer.name CRON[29440]: USER (good_user)
        #r=re.search(r'.*(\d\d:\d\d:\d\d).*USER \((\w+)\)$',line)
        r=re.search(r'.*(\d\d:\d\d:\d\d).*USER \((\w+)\)$',line)
        if r!=None:
          names[r[2]]=names.get(r[2],0) +1
          print(r[1],r[2])
print(names)

--------------------------------------

d={}

d['a']=d.get('a',0) + 1 if no value exist add 0 as default value and plus 1

Again, we're taking the current value in the dictionary by passing a default value of zero, so that when the key is in present in the dictionary, we had a default value. We then add one and set it as a new value associated with that key.

============-------

We're using the same syslog, and we want to display the date, time, and process id that's inside the square brackets. We can read each line of the syslog and pass the contents to the show_time_of_pid function. Fill in the gaps to extract the date, time, and process id from the passed line, and return this format: Jul 6 14:01:23 pid:29440.

pattern = r'([A-Z]\w\w \d \d\d:\d\d:\d\d) .*\[(\d{5})\].*'

return result[1]+' pid:'+result[2]


 

----------------------------------------------------

Obtaining the Output of a System Command

>>> import subprocess

>>> r=subprocess.run(['host','8.8.8.8'],capture_output=True)

>>> r.stdout

b'8.8.8.8.in-addr.arpa domain name pointer dns.google.\n'

>>> 'google' in r.stdout.decode()

True

------------------------------------------------------

Running System Commands in Python

>>> import subprocess                         
>>> subprocess.run(['date'])                 
Thu Sep  7 03:27:02 UTC 2023                  
CompletedProcess(args=['date'], returncode=0)                

>> subprocess.run(['sleep','5'])                                      
CompletedProcess(args=['sleep', '5'], returncode=0)

REGEX 
------------------------re-----------------------------------

Working with Regular Expressions  QWICK LAB

Introduction

It's time to put your new skills to the test! In this lab, you'll have to find the users using an old email domain in a big list using regular expressions. To do so, you'll need to write a script that includes:

  • Replacing the old domain name (abc.edu) with a new domain name (xyz.edu).
  • Storing all domain names, including the updated ones, in a new file.

You'll have 90 minutes to complete this lab.

Start the lab

Replace <csv_file_location> by the path to the user_emails.csv. <csv_file_location> is similar to the path /home/<username>/data/user_emails.csv. For variable report_file, replace <data_directory> by the path to /data directory. <data_directory> is similar to the path /home/<username>/data. Replace <username> with the one mentioned in the Connection Details Panel on the left-hand side.
#!/usr/bin/env python3
import re
import csv
def contains_domain(address, domain):
  """Returns True if the email address contains the given,domain,in the domain position, false if not."""
  domain = r'[\w\.-]+@'+domain+'$'
  if re.match(domain,address):
    return True
  return False
def replace_domain(address, old_domain, new_domain):
  """Replaces the old domain with the new domain in the received address."""
  old_domain_pattern = r'' + old_domain + '$'
  address = re.sub(old_domain_pattern, new_domain, address)
  return address
def main():
  """Processes the list of emails, replacing any instances of the old domain with the new domain."""
  old_domain, new_domain = 'abc.edu', 'xyz.edu'
  csv_file_location = '<csv_file_location>'
  report_file = '<path_to_home_directory>' + '/updated_user_emails.csv'
  user_email_list = []
  old_domain_email_list = []
  new_domain_email_list = []
  with open(csv_file_location, 'r') as f:
    user_data_list = list(csv.reader(f))
    user_email_list = [data[1].strip() for data in user_data_list[1:]]
    for email_address in user_email_list:
      if contains_domain(email_address, old_domain):
        old_domain_email_list.append(email_address)
        replaced_email = replace_domain(email_address,old_domain,new_domain)
        new_domain_email_list.append(replaced_email)
    email_key = ' ' + 'Email Address'
    email_index = user_data_list[0].index(email_key)
    for user in user_data_list[1:]:
      for old_domain, new_domain in zip(old_domain_email_list, new_domain_email_list):
        if user[email_index] == ' ' + old_domain:
          user[email_index] = ' ' + new_domain
  f.close()
  with open(report_file, 'w+') as output_file:
    writer = csv.writer(output_file)
    writer.writerows(user_data_list)
    output_file.close()
main()

------------------------------

TEST QUESTIONS REGEX AND RE.SEARCH RE.SUB

import re
def convert_phone_number(phone):
  result = re.sub(r' (\d{3})-(\d{3}-\d{4})',r" (\1) \2",phone,count=1)
  return result

print(convert_phone_number("My number is 212-345-9999.")) # My number is (212) 345-9999.
print(convert_phone_number("Please call 888-555-1234")) # Please call (888) 555-1234
print(convert_phone_number("123-123-12345")) # 123-123-12345
print(convert_phone_number("Phone number of Buckingham Palace is +44 303 123 7300")) # Phone number of Buckingham Palace is +44 303 123 7300
SOURCE CODE ONLY AGAIN AS UNDER
import re
def transform_record(record):
  new_record = re.sub("([\d-]+)",r"+1-\1" ,record)
  return new_record

print(transform_record("Sabrina Green,802-867-5309,System Administrator")) 
# Sabrina Green,+1-802-867-5309,System Administrator

print(transform_record("Eli Jones,684-3481127,IT specialist")) 
# Eli Jones,+1-684-3481127,IT specialist

print(transform_record("Melody Daniels,846-687-7436,Programmer")) 
# Melody Daniels,+1-846-687-7436,Programmer

print(transform_record("Charlie Rivera,698-746-3357,Web Developer")) 
# Charlie Rivera,+1-698-746-3357,Web Developer

--------------------------------------------------------------------------------------------------------

Positive Lookbehind: (?<=)

For example, we want to select the price value in the text. Therefore, to select only the number values that are preceded by $, we need to write the positive lookbehind expression (?<=) before our expression. Add \$ after the = sign inside the parenthesis.

Product Code: 1064 Price: $5
Report a Problem

Negative Lookahead: (?!)

For example, we want to select numbers other than the hour value in the text. Therefore, we need to write the negative look-ahead (?!) expression after our expression to select only the numerical values that do not have PM after them. Include PM after the ! sign inside the parentheses.

Date: 4 Aug 3PM
Report a Problem
-----------------------------------

Positive Lookahead: (?=)

For example, we want to select the hour value in the text. Therefore, to select only the numerical values that have PM after them, we need to write the positive look-ahead expression (?=) after our expression. Include PM after the = sign inside the parentheses.

Date: 4 Aug 3PM
Report a Problem
-------------------------------
Repetitions: Curly Braces
Practice: Curly Braces - 1
Write the expression using curly braces {} that will find texts containing 4 numbers side by side. Remember that the range [0-9] will match a single digit.

Release 10/9/2021
\d{4}   \d is digit means 0-9
[0-9]{4}    4  is 4 times
---------------------------------
+    1 or more mathcing      "he.*o"
?    0 or 1 occurance            "he.?o"
*    0 or many occurance     "he.*o"
-----------------------------------------

txt='br ber beer'

print(re.findall('be*r',txt))
['br', 'ber', 'beer']
* after any characher means that charchter (e) zero to many occurance means either no occurance or 1 to many times ocurrance.
so it also select br as and beer even it there we were beeeeeeeeeeeeer

------------------------------------------------

        print(re.search(r'A.*a','Argentenia'))

<re.Match object; span=(0, 10), match='Argentenia'>

print(re.search(r'A.*a','Azerbaijan'))
<re.Match object; span=(0, 9), match='Azerbaija'>

print(re.search(r'^A.*a$','Argentenia'))
<re.Match object; span=(0, 10), match='Argentenia'>

print(re.search(r'A.*a','Azerbaijan'))
<re.Match object; span=(0, 9), match='Azerbaija'>

print(re.search(r'A.*a$','Azerbaijan'))
None

-------------------------------

Fill in the code to check if the text passed looks like a standard sentence, meaning that it starts with an uppercase letter, followed by at least some lowercase letters or a space, and ends with a period, question mark, or exclamation point. 

Here is your output:
True
False
False
False
True

Awesome! You're becoming a regular "regular expression"
writer!

--------------------------------------------------------------------

QUESTION

Fill in the code to check if the text passed has at least 2 groups of alphanumeric characters (including letters, numbers, and underscores) separated by one or more whitespace characters.

Here is your output:
False
True
True
False

You got it! There's no escaping your regular expression
expertise!

----------------------RE Regular expression------

RegEx Functions

Taken from https://www.w3schools.com/python/python_regex.asp

The re module offers a set of functions that allows us to search a string for a match:

FunctionDescription
findallReturns a list containing all matches
searchReturns a Match object if there is a match anywhere in the string
splitReturns a list where the string has been split at each match
subReplaces one or many matches with a string

Metacharacters

Metacharacters are characters with a special meaning:

CharacterDescriptionExampleTry it
[]A set of characters"[a-m]"Try it »
\Signals a special sequence (can also be used to escape special characters)"\d"Try it »
.Any character (except newline character)"he..o"Try it »
^Starts with"^hello"Try it »
$Ends with"planet$"Try it »
*Zero or more occurrences"he.*o"Try it »
+One or more occurrences"he.+o"Try it »
?Zero or one occurrences"he.?o"Try it »
{}Exactly the specified number of occurrences"he.{2}o"Try it »
|Either or"falls|stays"Try it »
()Capture and group  

Special Sequences

A special sequence is a \ followed by one of the characters in the list below, and has a special meaning:

CharacterDescriptionExampleTry it
\AReturns a match if the specified characters are at the beginning of the string"\AThe"Try it »
\bReturns a match where the specified characters are at the beginning or at the end of a word
(the "r" in the beginning is making sure that the string is being treated as a "raw string")		r"\bain"
r"ain\b"		Try it »
Try it »
\BReturns a match where the specified characters are present, but NOT at the beginning (or at the end) of a word
(the "r" in the beginning is making sure that the string is being treated as a "raw string")		r"\Bain"
r"ain\B"		Try it »
Try it »
\dReturns a match where the string contains digits (numbers from 0-9)"\d"Try it »
\DReturns a match where the string DOES NOT contain digits"\D"Try it »
\sReturns a match where the string contains a white space character"\s"Try it »
\SReturns a match where the string DOES NOT contain a white space character"\S"Try it »
\wReturns a match where the string contains any word characters (characters from a to Z, digits from 0-9, and the underscore _ character)"\w"Try it »
\WReturns a match where the string DOES NOT contain any word characters"\W"Try it »
\ZReturns a match if the specified characters are at the end of the string"Spain\Z"Try it »

Sets

A set is a set of characters inside a pair of square brackets [] with a special meaning:

SetDescriptionTry it
[arn]Returns a match where one of the specified characters (a, r, or n) is presentTry it »
[a-n]Returns a match for any lower case character, alphabetically between a and nTry it »
[^arn]Returns a match for any character EXCEPT a, r, and nTry it »
[0123]Returns a match where any of the specified digits (0, 1, 2, or 3) are presentTry it »
[0-9]Returns a match for any digit between 0 and 9Try it »
[0-5][0-9]Returns a match for any two-digit numbers from 00 and 59Try it »
[a-zA-Z]Returns a match for any character alphabetically between a and z, lower case OR upper caseTry it »
[+]In sets, +, *, ., |, (), $,{} has no special meaning, so [+] means: return a match for any + character in the stringTry it »

Question

The repeating_letter_a function checks if the text passed includes the letter "a" (lowercase or uppercase) at least twice. For example, repeating_letter_a("banana") is True, while repeating_letter_a("pineapple") is False. Fill in the code to make this work.

import re
def repeating_letter_a(text):
  result = re.search(r"a.*a.*", text,re.IGNORECASE)
  return result != None

print(repeating_letter_a("banana")) # True
print(repeating_letter_a("pineapple")) # False
print(repeating_letter_a("Animal Kingdom")) # True
print(repeating_letter_a("A is for apple")) # True
Here is your output:
True
False
True
True

You get an A! See how handy the repetition qualifiers can
be, when we're working with lots of different text!
-----------------------------------------------------
txt = "The rain in Spain"
print(re.search("^The.*Spain$", txt))
<re.Match object; span=(0, 17), match='The rain in Spain'>
^The means text start with The
.* any character sequience 
Spain$ it would be at the last
---------------------------------------------------------------
print(re.search('P*','Python is programming'))
<re.Match object; span=(0, 1), match='P'>

print(re.search('P.*','Python is programming'))
<re.Match object; span=(0, 21), match='Python is programming'>

print(re.search('P.*n','Python is programming'))
<re.Match object; span=(0, 20), match='Python is programmin'>

print(re.search('P[a-z]*n','Python is programming'))
<re.Match object; span=(0, 6), match='Python'>
-----------------------

re.seach find only 1 and 1st occurance but RE.FINDALL seaches all
matching searches.
print(re.findall('cat|dog','I like both dogs and cats')
['dog','cat']
--------------------------------------------------------------
want to search either cat OR dog we have to use pipe |

| pipe symbol

r=re.search('cat|dog','I like cats')

out put wil be like 

<re.Match object; span=(7, 10), match='cat'>

-----------------------------------------------------

we want sometime must not be in a text group for we use

circumflex ^ inside the brakets [] like

[^]

r=re.search('[^a-zA-Z]','This is sentences with spaces.')

output will be <re.Match object; span=(4, 5), match='  '>

means at 4the index (start from 0) find a space character which is not

in [a-zA-Z]

if do not want even space match then we will add space in 

[a-zA-Z] as [a-zA-Z  ]

r=re.search('^a-zA-Z ],'This is sentence is with spaces.')

out put will be <re.Match object; span=(29, 30), match='.'>

----------------------------------------

import os

result=re.search(r'aza','plaza')

print(result)

<re.Match object; span=(2, 5), match='aza'>

result=re.search(r'aza','bazaar')

<re.Match object; span=(1, 4), match='aza'>

r=re.search(r'aza','saeed')

r

print(r)

None

result=re.search(r'^x)

To match any punctuation in the text
r=re.search([;.,!:?],text)

       
print(re.search(r'A.*a','Argentenia'))
<re.Match object; span=(0, 10), match='Argentenia'>

-------------------------------

grep command linux 

$ grep ^fruits /home/user/data/words  ^ is caret / circumflex

fruitscake

fruitsfull

fruitsed





Using the terminal, which of the following commands will correctly use grep to find the words “sling” and “sting” (assuming they are in our file, file.txt)?

Correct

Nice work! In regex, a dot is a wildcard, so it can represent any character. This command will print both “sting” and “sling”, if they are in the file.

import re

Rather than using the index() function of the string module, we can use regular expressions, which are more flexible. After importing the regular expression module re, what regex function might be used instead of standard methods?

Correct

Great job! Using the re module provides more robust solutions, not limited to just re.search().

------------------------------------------------------------

I made following project from my brain as under: a simple approach

import csv
with open('employees.csv') as normalfile:
    # open file as csv.DictReader()
    csvfile = csv.DictReader(normalfile)
    depts=[]
    for row in csvfile:
        i+=1
        depts.append(row['Department'] )
setdepts =  set(depts)
for dept in setdepts:
    TOTAL=depts.count(dept)
    print(dept,TOTAL)


Option 1: Windows Users: Connecting to your VM

In this section, you will use the PuTTY Secure Shell (SSH) client and your VM’s External IP address to connect.

Download your PPK key file

You can download the VM’s private key file in the PuTTY-compatible PPK format from the Qwiklabs Start Lab page. Click on Download PPK.

Connect to your VM using SSH and PuTTY

  1. You can download Putty from here

  2. In the Host Name (or IP address) box, enter username@external_ip_address.

Putty_1


  1. In the Connection list, expand SSH.

  2. Then expand Auth by clicking on + icon.

  3. Now, select the Credentials from the Auth list.

  4. In the Private key file for authentication box, browse to the PPK file that you downloaded and double-click it.

  5. Click on the Open button.

Putty_2

  1. Click Yes when prompted to allow a first connection to this remote SSH server. Because you are using a key pair for authentication, you will not be prompted for a password.

Common issues

If PuTTY fails to connect to your Linux VM, verify that:

  • You entered <username>@<external ip address> in PuTTY.

  • You downloaded the fresh new PPK file for this lab from Qwiklabs.

  • You are using the downloaded PPK file in PuTTY.

Prerequisites

We have created the employee list for you. Navigate to the data directory using the following command:

cd data
Copied!

To find the data, list the files using the following command:

ls
Copied!

You can now see a file called employees.csv, where you will find your data. You can also see a directory called scripts. We will write the python script in this directory.

To view the contents of the file, enter the following command:

cat employees.csv
Copied!

Let's start by writing the script. You will write this python script in the scripts directory. Go to the scripts directory by using the following command:

cd ~/scripts
Copied!

Create a file named generate_report.py using the following command:

nano generate_report.py
Copied!

You will write your python script in this generate_report.py file. This script begins with a line containing the #! character combination, which is commonly called hash bang or shebang, and continues with the path to the interpreter. If the kernel finds that the first two bytes are #! then it uses the rest of the line as an interpreter and passes the file as an argument. We will use the following shebang in this script:

#!/usr/bin/env python3

Convert employee data to dictionary

The goal of the script is to read the CSV file and generate a report with the total number of people in each department. To achieve this, we will divide the script into three functions.

Let's start with the first function: read_employees(). This function receives a CSV file as a parameter and returns a list of dictionaries from that file. For this, we will use the CSV module.

The CSV module uses classes to read and write tabular data in CSV format. The CSV library allows us to both read from and write to CSV files.

Now, import the CSV module.

import csv
Copied!

Define the function read_employees. This function takes file_location (path to employees.csv) as a parameter.

def read_employees(csv_file_location):
Copied!

Open the CSV file by calling open and then csv.DictReader.

DictReader creates an object that operates like a regular reader (an object that iterates over lines in the given CSV file), but also maps the information it reads into a dictionary where keys are given by the optional fieldnames parameter. If we omit the fieldnames parameter, the values in the first row of the CSV file will be used as the keys. So, in this case, the first line of the CSV file has the keys and so there's no need to pass fieldnames as a parameter.

We also need to pass a dialect as a parameter to this function. There isn't a well-defined standard for comma-separated value files, so the parser needs to be flexible. Flexibility here means that there are many parameters to control how csv parses or writes data. Rather than passing each of these parameters to the reader and writer separately, we group them together conveniently into a dialect object.

Dialect classes can be registered by name so that callers of the CSV module don't need to know the parameter settings in advance. We will now register a dialect empDialect.

  csv.register_dialect('empDialect', skipinitialspace=True, strict=True)
Copied!

The main purpose of this dialect is to remove any leading spaces while parsing the CSV file.

The function will look similar to:

  employee_file = csv.DictReader(open(csv_file_location), dialect = 'empDialect')
Copied!

You now need to iterate over the CSV file that you opened, i.e., employee_file. When you iterate over a CSV file, each iteration of the loop produces a dictionary from strings (key) to strings (value).

Append the dictionaries to an empty initialised list employee_list as you iterate over the CSV file.

  employee_list = []
  for data in employee_file:
    employee_list.append(data)
Copied!

Now return this list.

  return employee_list
Copied!

To test the function, call the function and save it to a variable called employee_list. Pass the path to employees.csv as a parameter to the function. Print the variable employee_list to check whether it returns a list of dictionaries.

employee_list = read_employees('<file_location>')
print(employee_list)
Copied!

Replace <file_location> with the path to the employees.csv (this should look similar to the path /home/<username>/data/employees.csv). Replace <username> with the one mentioned in Connection Details Panel at left hand side.

Save the file by clicking Ctrl-o, Enter, and Ctrl-x.

For the file to run it needs to have execute permission (x). Let's update the file permissions and then try running the file. Use the following command to add execute permission to the file:

chmod +x generate_report.py
Copied!

Now test the function by running the file using the following command:

./generate_report.py
Copied!

The list employees_list within the script should return the list of dictionaries as shown below.

67583834df5977bb.png

Click Check my progress to verify the objective.

Convert employee data to dictionary

Process employee data

The second function process_data() should now receive the list of dictionaries, i.e., employee_list as a parameter and return a dictionary of department:amount.

Open the file generate_report.py to define the function.

nano generate_report.py
Copied!

def process_data(employee_list):
Copied!

This function needs to pass the employee_list, received from the previous section, as a parameter to the function.

Now, initialize a new list called department_list, iterate over employee_list, and add only the departments into the department_list.

  department_list = []
  for employee_data in employee_list:
    department_list.append(employee_data['Department'])
Copied!

The department_list should now have a redundant list of all the department names. We now have to remove the redundancy and return a dictionary. We will return this dicationary in the format department:amount, where amount is the number of employees in that particular department.

  department_data = {}
  for department_name in set(department_list):
    department_data[department_name] = department_list.count(department_name)
  return department_data
Copied!

This uses the set() method, which converts iterable elements to distinct elements.

Now, call this function by passing the employee_list from the previous section. Then, save the output in a variable called dictionary. Print the variable dictionary.

dictionary = process_data(employee_list)
print(dictionary)
Copied!

Save the file by clicking Ctrl-o, Enter, and Ctrl-x.

Now test the function by running the file using the following command:

./generate_report.py
Copied!

This should return a dictionary in the format department: amount, as shown below.

1c2daf5cbaa58c07.png

Click Check my progress to verify the objective.

Process employee data

Generate a report


Next, we will write the function write_report. This function writes a dictionary of department: amount to a file.

The report should have the format:

<department1>: <amount1>

<department2>: <amount2>

Lets open generate_report.py file to define the function.

nano generate_report.py
Copied!

def write_report(dictionary, report_file):
Copied!

This function requires a dictionary, from the previous section, and report_file, an output file to generate report, to both be passed as parameters.

You will use the open() function to open a file and return a corresponding file object. This function requires file path and file mode to be passed as parameters. The file mode is 'r' (reading) by default, so you should now explicitly pass 'w+' mode (open for reading and writing, overwriting a file) as a parameter.

Once you open the file for writing, iterate through the dictionary and use write() on the file to store the data.

  with open(report_file, "w+") as f:
    for k in sorted(dictionary):
      f.write(str(k)+':'+str(dictionary[k])+'\n')
    f.close()
Copied!

Now call the function write_report() by passing a dictionary variable from the previous section and also passing a report_file. The report_file passed within this function should be similar to /home/<username>/data/report.txt. Replace <username> with the one mentioned in Connection Details Panel at left-hand side.

write_report(dictionary, '<report_file>')
Copied!

Save the file by clicking Ctrl-o, Enter, and Ctrl-x.

Let's execute the script now.

./generate_report.py
Copied!

This script does not generate any output, but it creates a new file named report.txt within the data directory. This report.txt file should now have the count of people in each department.

Navigate to the data directory and list the files. You should see a new file named report.txt.

cd ~/data
Copied!

ls
Copied!

To view the generated report file, use the following command:

cat report.txt
Copied!

The report file should be similar to the below image.

6414b93d8fccda7b.png

Click Check my progress to verify the objective.

Generate a report

Congratulations!

You successfully wrote a Python script that achieves two tasks. First, it reads a CSV file containing a list of the employees in the organization. Second, it generates a report of the number of people in each department in a plain text file.

Creating reports using Python is a very useful tool in IT support. You will likely complete similar tasks regularly throughout your career, so feel free to go through this lab more than once. Remember, practice makes perfect.

End your lab

When you have completed your lab, click End Lab. Qwiklabs removes the resources you’ve used and cleans the account for you.

You will be given an opportunity to rate the lab experience. Select the applicable number of stars, type a comment, and then click Submit.

---------------------------------------------

Qwiklabs Assessment: Handling Files

Introduction

For this lab, imagine you are an IT Specialist at a medium-sized company. The Human Resources Department at your company wants you to find out how many people are in each department. You need to write a Python script that reads a CSV file containing a list of the employees in the organization, counts how many people are in each department, and then generates a report using this information. The output of this script will be a plain text file.

----------------KWI

How to Log in to Qwiklabs

In the following assessments, you’ll be using Qwiklabs for hands-on learning. Qwiklabs provisions resources backed by Google Cloud that will be used to perform the tasks in the assessments. By using Qwiklabs, you won't have to purchase or install software yourself, and you can use the Linux operating system as if it was installed on your local machine.

Important details:

  • You will have 90 minutes to complete each lab.

  • You'll experience a delay as the labs load, as well as for the working instances of Linux VMs. So, please wait a couple of minutes.

  • Make sure to access labs directly through Coursera and not in the Qwiklabs catalog. If you access labs through the Qwiklabs catalog, you will not receive a grade. (As you know, a passing grade is required to complete the course.)

  • You'll connect to a new VM for each lab with temporary credentials created for you; these will last only for the duration of the lab.

  • The grade is calculated when the lab is complete, so be sure to hit "End Lab" when you're done. Note: after you end the lab, you won't be able to access your previous work.

  • To get familiar with entering labs, find the links below for the operating system of the machine you are currently using for a visualization of the key steps. Note that while video resources linked below do not have a voiceover or any audio, all important details will still be housed in each lab’s set of instructions on the Qwiklabs platform.

  • If you receive the error "Sorry, your quota has been exceeded for the lab", please submit a request or reach out to the Qwiklabs support team directly via chat support on qwiklabs.com.


-------------------------------------------------------------------------

What is a virtual machine?

Correct

Nice job! A virtual machine, or VM, is a computer simulated through software. It simulates all the necessary hardware to the operating system that's running inside the machine.

----------------------------------------------------------------------











-----------------------------------------------------------------



---------------------------------------------------

 with open(filename) as normalfile:
    # Read the rows of the file into a dictionary
    csvdictfile = csv.DictReader(normalfile)
    for row in scvdictfile:
        print(row)
        print(row['name'])

---------------------------------------------

F:\0 Python>type temp.csv
Name,Age,Location
John,25,New York
Alice,30,Los Angeles
Bob,22,Chicago

To read above file using csv.DictReader
# first open file normal with read only
with open('temp.csv') as normalfile:
    csvfile=csv.DictReader(normalfile)
    for row in csvfile:
        print(row)
OUT PUT 
{'Name': 'John', 'Age': '25', 'Location': 'New York'}
{'Name': 'Alice', 'Age': '30', 'Location': 'Los Angeles'}
{'Name': 'Bob', 'Age': '22', 'Location': 'Chicago'}

-------------------------------------------

csv.DictReader() is also a very good Handy function

and also DictWriter()  for making Dict csv file

---------------------------------------------

import csv

# Sample list of data get from chatGPT

data_list = [

    ['Name', 'Age', 'Location'],

    ['John', 25, 'New York'],

    ['Alice', 30, 'Los Angeles'],

    ['Bob', 22, 'Chicago']

]

# open file normal as 'w' permissions

with open('temp.csv','w',newline='') as normalfile:

  csvfile = csv.writer(normalfile)

  # first thing that you will addrows() not addrows but is writerows()

  # addrows() not addrows but writerows()

  # First question will be arose the where are the rows/list to be added

  # that are om data_list so we will write

  # writerows(data_list)

  # Second Question in which we writerows() Answer will in csvfile

  csvfile.writerows(data_list) # csvfile is already made by above statement BUS (STOP in urdu/hindi), STOP

with open('temp.csv') as f:

  csvfile = csv.DictReader(f)

  #print(csvfile.fieldnames())

  for row in csvfile:

    print(row)

    print(row["Name"],row["Age"],row["Location"])

-----------------------------------

For writing in or making new csv file

f=open('file.csv','w')

f.csv.writerows(emp)

For reading csv file


----------------------------------------

Which of the following must we do before using the csv.writer() function?


Correct

Nice work! The file must be open, preferably using with open() as, and write permissions must be given.


--------------------------------------------------------

Which of the following lines would correctly interpret a CSV file called "file" using the CSV module? Assume that the CSV module has already been imported.

-------------------------------------------------------------------

PARSING : is analysing the contents of a file to collect correct data structure 

if we have data in a format we understand, then we have what we need to parse the information from the file. What does parsing really mean?

-----------------------------------------------------------------------------

some OS module functions used

import os

os.remove(mfilename)

os.rename(oldfile,newfile)

os.path.exists('saeed.txt') if file exist return True / False

os.path.getsize(filename) to check files byte size


os.path.getmtime('temp.py')   get modified time O/P will be as under

1693390458.2565691   its a timestamp

timestamp=os.path.getmtime('saeed.txt')  m=Modified time

import datetime

datetime.datetime.fromtimestamp(timestamp)

out put will be some thing like

(2023, 8, 30, 15, 14, 18, 256569)

os.path.abspath('temp.txt') give absolute path these types of commands os command can be run is each and every OS like Linux, Unix, Window or Apple Os  or Andriod etc. 

the path will be given according to each OS No need to recode for each OS. The out put of 

command in Windows is like:

f:\\0 Python\\temp.py' Note that one slash is another meaning like in print command \n or \t

if you want to print \ your have to write print('\\')  not print('\') it will give syntex error 

if run same command 

os.path.abspath('saeed.txt') in Linux then out put will be some thing like

/home/user/saeed.txt

this is the power of Python a modern language Once you write code it will be platform (Linux, window, apple OS, Andriod) independent.

os.mkdir('saeed')

os.chdir('saeed')

os.getcwd()   Get Current Working |Directory

os.rmdir('saeed.txt')

To check is it directory or File  os.path.isdir('temp.txt')

--------------------------------------------------------

Never use absolute path of file in your program Make the program general type so that is may run on all the OSes like Linux , Windows. So use the 

import os

os.path functions 

like os.path.getcwd()

os.path.chdir

path=os.path.join(os.path.getcwd(),mfile_name)

join function os.path.join(dirname,dirname)

mdir=os.path.join(os.getcwd(),'mdir1','mdir2','mdir3')

the out in windows systems is like

'F:\\0 Python\\mdir1\\mdir2\\mdir3'

if same command is given in Linux then out put would be same thing lik

'/home/user/0 Python/mdir1/mdir2/mdir3'

It means the coding will be plateform independent 

Question

What's the purpose of the os.path.join function?

Files and Directories Cheat-Sheet

Check out the following links for more information:

Other eg which I made
try:
    print('To check how all errors are by passed')
    this is error
except:
    pass
In result all errors are by passed

============================================================

Week 1 is only some bawas, Python installation, python file.py runing from Os.

with open('file.txt') as f:

after working on file it will automatically closes the file

=============================

read file line by line , to save the memory of computer.

do not use mlines=file.readlines() means read all lines of file in memory var mlines it uses the and occupy the memory of computer

====================================

what is Jupyter Notebook


Comments

Post a Comment

Popular posts from this blog

PANDAS micro course by www.Kaggle.com https://www.kaggle.com/learn/pandas

Image
  Creating, Reading and Writing $ pip install pandas $ python3 >>> import pandas as pd >>> pd.DataFrame({'BOB':['Ist column of BOB','IInd col of BoB'],'SUSE':['Ist col suse','II col suse']})                  BOB          SUSE 0  Ist column of BOB  Ist col suse 1    IInd col of BoB   II col suse >>>pd.DataFrame({..same as above---},index=['Prouct a ',' Product b'])   >>> pd.Series([30, 35, 40], index=['2015 Sales', '2016 Sales', '2017 Sales'], name='Product A') 2015 Sales    30 2016 Sales    35 2017 Sales    40 Name: Product A, dtype: int64 >>> reviews=pd.read_csv('/mnt/d/wine.csv') >>> reviews show the contents of file wine.csv >>> reviews.country >>> reviews['country']   to see only country column >>> reviews.iloc[0] return only one row >>> review.il...
Read more

Introduction to Git and GitHub https://www.coursera.org/learn/introduction-git-github/

Image
Introduction to Git and GitHub https://www.coursera.org/learn/introduction-git-github/ On top of search results, here are some great Git resources available online: Pro Git : This book (available online and in print) covers all the fundamentals of how Git works and how to use it. Refer to it if you want to learn more about the subjects that we cover throughout the course. Git tutorial : This tutorial includes a very brief reference of all Git commands available. You can use it to quickly review the commands that you need to use. WEEK NO 1  Intro to Module 1: Version Control First I have to install Git in each computer???????????!!!!!!!!!!!! https://git-scm.com/download/win Instead of new coding we must use the previous codes and extend/add  functions/codes because previous codes are alrady tested from bugs and are debugged .  in VCS/Git there are two main utilities  DIFFand PATCH After installatin of GIT we can track the changes in codes. Week 1 What is version cont...
Read more