Please click on the following link to view the poster and meta-scheduling.pptx.
Poster
meta-scheduling.pptx
Monday, March 7, 2011
Wednesday, February 23, 2011
MURPA-Lin Wei-Week 7
1. User Interface of OpalNimrod actor
In week 7, I implemented the interface of OpalNimrod actor. The screen shot below shows that users select application name from a drop down list. From the users' point of view, it is more convenient than manually defining an URL because they don't need to know the details of any service URL. The actor obtains the service URLs and host meta-data from Opal registry.
A dummy xml file which represents the Opal registry was uploaded to NBCR Wiki. OpalNimrod actor access this xml to retrieve service URLs and number of free CPUs. The real Opal registry may be implemented in a database in the future. But for the purpose of this project and simplicity reason, a dummy xml file is sufficient.
2. Using Nimrod/K Scheduler in OpalNimrod Actor
I'm thinking about creating an instance of the NimrodResource class in OpalNimrod actor to interact with the Opal registry to select a service URL. Unfortunately, the NimrodResource class assigns an actor to resource with available slots rather than an URL. So an OpalResource class needs to be created to return a service URL to the job submission side of the OpalNimrod actor. Since the OpalResource class is only useful to OpalNimrod actor, I plan to write it as a nested class in the actor.
3. Autodock workflow
For unknown reasons, the Opal actor doesn't work with Sequence to Tag actor to concurrently execute Autodock application in Kepler 1.0. However, the Autodock workflow works well in Kepler 2.0 without Sequence to Tag actor. An Autodock workflow tutorial was uploaded to NBCR WiKi Kepler page.
In week 7, I implemented the interface of OpalNimrod actor. The screen shot below shows that users select application name from a drop down list. From the users' point of view, it is more convenient than manually defining an URL because they don't need to know the details of any service URL. The actor obtains the service URLs and host meta-data from Opal registry.
A dummy xml file which represents the Opal registry was uploaded to NBCR Wiki. OpalNimrod actor access this xml to retrieve service URLs and number of free CPUs. The real Opal registry may be implemented in a database in the future. But for the purpose of this project and simplicity reason, a dummy xml file is sufficient.2. Using Nimrod/K Scheduler in OpalNimrod Actor
I'm thinking about creating an instance of the NimrodResource class in OpalNimrod actor to interact with the Opal registry to select a service URL. Unfortunately, the NimrodResource class assigns an actor to resource with available slots rather than an URL. So an OpalResource class needs to be created to return a service URL to the job submission side of the OpalNimrod actor. Since the OpalResource class is only useful to OpalNimrod actor, I plan to write it as a nested class in the actor.
3. Autodock workflow
For unknown reasons, the Opal actor doesn't work with Sequence to Tag actor to concurrently execute Autodock application in Kepler 1.0. However, the Autodock workflow works well in Kepler 2.0 without Sequence to Tag actor. An Autodock workflow tutorial was uploaded to NBCR WiKi Kepler page.
Tuesday, February 15, 2011
MURPA-Lin Wei-Week 6
In this week, Wilfred agrees that we should build an OpalNimrod actor. The steps required to implement a prototype of the OpalNimrod actor:
1. Create a dummy xml file that stores application metadata.
2. Create a OpalNimrod actor based on the existing Opal actor.
3. The OpalNimrod actor uses Java Document Object Model to access the xml file.
4. Create a new user interface for the OpalNimrod to allow users to select an application name from a drop down list instead of defining the service URL. The drop down list is dynamically generated from the dummy xml file. After users select an application and click the commit button, the interface should display the input parameters for the selected application.
5. After users specify the parameter values and run the workflow, all the serviceURLs that run the application are stored in an array along with the serviceURL metadata. The array is passed to the Nimrod/K scheduler. The scheduler chooses a serviceURL based on the number of free CPUs.
6. At last, the input parameters are submitted to the chosen service URL.
The actor can be tested by changing the value of the number of free CPUs in the XML file and run a PDB2PQR workflow. ws.nbcr.net and kryptonite are the stable hosts that provide PDB2PQR web service. The Nimrod scheduler should choose between these two hosts.
Apart from creating an OpalNimrod actor, I will make a tutorial on how to run an Autodock virtual screening workflow on the NBCR wiki page because the new Opal plug-in will be released soon.
1. Create a dummy xml file that stores application metadata.
2. Create a OpalNimrod actor based on the existing Opal actor.
3. The OpalNimrod actor uses Java Document Object Model to access the xml file.
4. Create a new user interface for the OpalNimrod to allow users to select an application name from a drop down list instead of defining the service URL. The drop down list is dynamically generated from the dummy xml file. After users select an application and click the commit button, the interface should display the input parameters for the selected application.
5. After users specify the parameter values and run the workflow, all the serviceURLs that run the application are stored in an array along with the serviceURL metadata. The array is passed to the Nimrod/K scheduler. The scheduler chooses a serviceURL based on the number of free CPUs.
6. At last, the input parameters are submitted to the chosen service URL.
The actor can be tested by changing the value of the number of free CPUs in the XML file and run a PDB2PQR workflow. ws.nbcr.net and kryptonite are the stable hosts that provide PDB2PQR web service. The Nimrod scheduler should choose between these two hosts.
Apart from creating an OpalNimrod actor, I will make a tutorial on how to run an Autodock virtual screening workflow on the NBCR wiki page because the new Opal plug-in will be released soon.
Tuesday, February 8, 2011
MURPA-Lin Wei-Week 5
In week 5, I spent the majority of my time fixing the opal.xml workflow, working on the OpalResource class design and studying the OpalClient.java. In terms of the workflow, it still doesn’t work. I have tried all the suggestions that were made by my supervisors and my own idea, but the error remains.
Opal.xml workflow
The actions I performed include:
| Action | Detailed Step | Results |
| Colin’s suggestion | 1. Install Nimrod/K and copy all the jar files that Colin gave me to Kepler 2. using this URL for the workflow: http://kryptonite.nbcr.net/app1267557427870/. Jane has double checked for me that this URL is a working URL. 3. On the workflow, update the Constant actor and update the all the "u" fields with this URL and update the "dpf" fields with the local copy (where kepler is) of 2HU4_B.dpf. 4. Run the workflow | ptolemy.kernel.util.IllegalActionException: We could not find the attribute filter_file |
| Wilfred’s suggestion | 1. Export the OpalClient.java in Kepler 2.0 as an OpalClient.jar file 2. Replace the OpalClient.jar in Kepler 1.0 with the new OpalClient.jar 3. Run the workflow with the URL and the dpf file. I’m not sure if these are the correct steps. But the error message indicates that the OpalClient.java may be different because the OpalClient._makeCmd are on different lines | ptolemy.kernel.util.IllegalActionException: We could not find the attribute filter_file |
| My idea | 1. Open the PDB2PQR.xml workflow that was created in Kepler 2.0 in Kepler 1.0 2. Open the opal.xml workflow in Kepler 1.0. 3. Copy the OpalClient actor in PDB2PQR workflow and paste it in opal.xml workflow to replace the original OpalClient actor. 4. Using the same serviceURL, u, dpf file and all other parameter values for the copied OpalClient actor. 5. Run the workflow and skip the error. | It is interesting that the workflow runs successfully with error messages flood the entire error message dialogue window. I showed Jane the generated web pages and she confirmed that the job was actually executed. |
It remains unknown that if the OpalClient actor has been modified and used in the opal.xml workflow. All people involved in the opal.xml workflow development have been consulted and no one could tell what exactly he/she did when created the workflow. However, I suspect that the OpalClient actors are different because the interfaces of the actors in Kepler 2.0 and Kepler 1.0 are different. Here are the screenshots of the interfaces:
OpalClient interface in Kepler1.0
OpalClient Interface in Kepler 2
OpalResource class Design
The OpalResource class design needs to be modified. Wilfred points out that if we build the OpalResource class inside the OpalClient actor, the Opal actor will be dependent on the Nimrod/K API. Due to copyright issues, the Nimrod/K does not come with the standard Kepler distribution. The users have to agree to the licence, download and install the Nimrod/K manually. Wilfred suggests create an OpalNimrod actor which does the same job as the OpalResource class, but it adds burdens to maintenance since we will end up with two actors. We will discuss the design issues further with Colin in week 6.
Spirit Night
We went out to watch basketball competition at UCSD on spirit night. I was amazed by the passionate students and atmosphere of the competition. It’s a pity that Monash doesn’t have such sports event.
Tuesday, February 1, 2011
MURPA-Lin Wei-Week 4
Progress of the project
During the Skype chat with Colin, I have been advised that Nimrod/K has both parameter sweep and meta-scheduling capabilities. Therefore, instead of deploying a scheduler or Nimrod/G in the Opal server to make scheduling decisions, we can use the existing scheduler inside the Nimrod/K. There are two reasons for this choice. On the one hand, researchers at MeSsAGE Lab are focusing on developing Nimrod/K, using Nimrod/K will leverage the new features and functions. On the other hand, the end users will have the freedom to configure the Nimrod/K actors in the Kepler to fulfil their specific resource requirements. For instance, the users can specify the number of CPUs and the size of the memory to be assigned to execute a job.
The initial design of how to use the Nimrod/K scheduler with Opal web services are detailed as follow:
Overview of the OpalResource Class
In order to utilize the Nimrod/K’s meta-scheduling capability, a new class called OpalResource needs to be introduced to the Opal actor. There are three major tasks that the OpalResource class handles.
1.It synchronizes the metadata (Number of totoal CPUs, number of free CPUs, Free Memory, etc) about the Opal resource with the Opal server. The Opal resource could be cluster, grid or cloud resources that runs scientific application. At the initial stage of the project, the scheduler makes allocation decision based on the number of CPUs required to execute a job.
2.It provides the Opal resource metadata to the scheduler.
3.It monitors and maintains the job submission and resource allocation request to the Opal server.
Both the scheduler and the Opal actor has a reference to the OpalResource class so that they can interacts with the Opal resource.
Creating OpalResource Instances
This diagram illustrates how OpalResource class creates instances of Opal resources. When end users click on the run button in Kepler to execute the Autodock workflow, it triggers the OpalReousr class. The OpalResource class requests for the metadata of the resources that can run the Autodock application. Since the Opal server has records of all the Autodock resources, it can simply fullfill the request by transferring the Autodock resource metadata to the OpalResource class. Then, for each Autodock resource, the OpalResource class creates an OpalResource instance to keep track of the resource metadata. It is important to notice that the entire process is syncronized to ensure that the resources metadata in the Opal server is identical to the ones in the OpalResource instances. For instance, if 10 CPUs becomes available in an Autodock resource, it is also reflected in its corresponding OpalResource instances.
Creating Copies of Opal Actor
Nimrod/K performs parameter sweep by generating multiple copies of the Opal actor. Each copy of the Opal actor is supplied with a set of parameters. It is up to the end user to define the number of parameter set and the value of parameters in each parameter set. The parameter sets are prepared in GirdTokenFiles. Each GridTokenFileis is tagged with a different color to distinguish themselves. Although the copies of the Opal actor are not shown in Kepler, these copies are created and running on the background.
Resource Allocation
Each copy of the Opal actor for Autodock experiment may have different resource requirements. The diagram takes one copy of the Autodock Opal actor to demonstrate how resources are allocated for job execution. First, the copy of the Opal actor provides the resource requirement to the OpalResource class. Then, the OpalResource class pass the resource requirement along with the references of the Autodock OpalResource instances to the scheduler. The scheduler calls the getFreeCPUs to access to Autodock OpalResource metadata and invokes getSlots(int) method to make the allocation decision. After the allocation decision has been made, it is passed back to the OpalResource class. At the end, the OpalResource class submits the Autodock job request along with the resource allocation decision to the Opal server. The Opal server will execute the job on the Autodock resources according to the allocation decision and notify the OpalResource class if the allocation request is fulfilled. Again, The communication in the entire resource allocation process is synchronized. But for the convenience of describing the process, it is broken down into steps.
Some Thoughts About the Opal.xml Workflow
What is Nimrod/K director’s dynamic parallelism?
Why the users have to type the input in an array in the Constant actor? Is there a more convenient way for users to provide input?
How sequences are converted tags? Why manually pre-stage files? Can we design a mechanism to automatically handle the files?
What does Expression actor do?
Can we get rid of the Array to Sequence actor and Expression actor so that the Nimrod/K and the Opal actor can interacts directly?
Saturday, January 22, 2011
MURPA-Lin Wei-Week 3
Progress of the project
1.The big picture of the project.
After playing with Kepler, reading documentation and talking with UCSD researchers, I found that there was a misunderstanding of the project. When Wilfred talked about integrating Opal and Nimrod/K originally, he actually wants to use Nimrod/K’s meta-scheduling capability to assign jobs to the available resources regardless of the geographical location of the resources. On the contrary, Colin and Ilkay thought that Wilfred wants to do parameter sweep with Nimrod/K and Opal actor. That’s why Colin suggested me to extend the Opal actor to use GridFileTokens, and Ilkay asked me to create work flow first. I have a strong feeling that the Nimrod/K may not do meta-scheduling, only the Nimrod itself has meta-scheduling capability. Even if it does, how could it make scheduling decisions when the Opal actor submits the requests to Opal? Nimrod/K simply does not know the state of the resources.
When I discussed the misunderstanding with Wilfred, both of us agreed that we need to confirm it with Colin during the Skype meeting on next Monday. However, there is no conflict between the parameter sweep and meta-scheduling. They are two separate components of the whole project. The picture below illustrates these two parts and the relationship between them. In the Kepler work flow environment, we can follow colin’s idea to use GridFileToken with Opal actor to implement parameter sweep. In terms of the meta-scheduling, we might be able to extract the meta-scheduling component of the Nimrod and implement it as a job manager in Opal.
Now, the question is which part should I work on for the next 5 weeks? I would recommend work on the meta-scheduling part because Colin is very knowledgeable about the Nimrod/K parameter sweep and another Monash student is upgrading Nimrod/K to version 2.X. The advantage of doing research in UCSD is that I can use Wilfred and his team’s expertise in Opal. Therefore, it might be a better idea to focus on the meta-scheduling part. But I still need to discuss this with Wilfred and Colin.
2. Fix the work flow empty output errors.
The original MEME work flow returns a web page without any DNA sequence analysis results. This problem is not caused by the design of the work flow. It is an issue with the Opal MEME web service.
This error was quickly fixed by changing the input file and command line argument.
New command: meme crp0.fasta -dna -nmotifs 3
New sequence file downloaded from:
http://meme.sdsc.edu/meme4_5_0/doc/examples/sample_opal_scripts/crp0.fasta
Work flow output can be accessed from: http://ws.nbcr.net/app1295647544139/meme.html
Other part of the work flow remains the same.
3. Study sample parameter sweep workflow
Jianwu sent me some parameter sweep work flows and a paper describing the use case of the work flow. Unfortunately, none of them uses Nimrod/K to do the parameter sweep. They use the parameter sweep actor in Kepler instead. These sample work flow are good start point for me to understand how parameter sweep is implemented.
Paper:
Since part of the project is to use Nimrod/K to handle parameter sweep, I would like to see some Nimrod/K work flow. I’m going to ask Colin for some Nimrod/K work flow in the Skype meeting.
Plan for the weekendsSan Diego Zoo and downtown
Wednesday, January 19, 2011
MURPA-Lin Wei-Week 2
Progress of the Project
It is challenging and interesting that my three supervisors have different ideas about the project. Wilfred is interested in using Nimrod's meta-scheduling capability. Ilkay wants to create a work flow that can be extended later to include parameter sweep. Colin suggested me to extend the Opal actor to use GridFileTokens. While those ideas add an extra level of complexity to the project, it helps me view the project from different perspectives because my supervisors came up with the idea based on their expertise.
In week 2 meeting, it has been finally agreed that we will follow Ilkay's suggestion to construct a work flow first. Thanks to Jianwu and Jane's assistance, I have two candidate work flows at the moment:
This work flow submits DNA or protein sequences to MEME. MEME will analyze your sequences for similarities among them and produce a description for each pattern/motif it discovers. The analysis results is displayed in a web browser. The work flow is retrieved from https://code.kepler-project.org/code/kepler/trunk/workflows/meme-opal/ To view a sample output, visit http://ws.nbcr.net/app1295390134402/. The required parameters to be specified in this work flow include the occurrences of a single motif, maximum and minimum width of motif, maximum number of motif. There are also many optional parameters. Those parameters can be used later to implement parameter sweep.
The original work flow use MEME 4.1.0 web service which is no longer supported by NBCR. Therefore, I modified the work flow to use MEME 4.5.0. The problem with the work flow is that when I save the modified work flow, the work flow cannot find At.fa sequence file at execution time. I plan to consult Jianwu again to fix this problem.
This work flow perform a PDB to PQR file format conversion using NBCR remote computation capabilities. It runs smoothly. Visit http://ws.nbcr.net/app1295392505925/ to view the results. However, I'm not sure if it make sense to do parameter sweep. I need to talk to Wilfred to find out if the parameter sweep can help solve scientific problem.
Entertainment
We rent a car and visited Las Vegas during the long weekends. We pretty much went to every big casino. Unfortunately, Wai Keung and Geoff lost around $100 in gambling. The buildings and scenery in downtown are more attractive to me than gambling.
Bad News
My dad told me that my mum had a big surgery. The surgeon opened my mum's belly and removed four tumors. I hope my mum will recover soon.
Monday, January 10, 2011
MURPA-Lin Wei-Week 1
Finally, Wai Keung and I arrived at San Diego on Tuesday. The long boring flight makes me exhausted. Not mentioned that the long queue at Los Angeles immigration kept us waiting for an hour and we missed the flight to San Diego. The security staff who inspected my visa was unfriendly. He never greets visitors and the way he looks at me makes me feel that he just wants to reject anyone from entering the United States.
The weather at San Diego in winter is so pleasant, the apartment is fantastic and the people at UCSD are friendly. Everything at UCSD helps me recover very soon and I feel I'm in the right mode to work on my project. I particularly like the supermarket here. It's still open after midnight and it's got tons of food, much better than Australian supermarkets. The plastic bag design is brilliant. There is no need to rub the bag to open it. You pull the front bag and the next bag opens automatically. This design reminds me Don Normans's book "The design of everyday things". This book is the best introduction to the importance of usability in design: much of the value comes from the fact that it is not about computers but about all kinds of other things that we suffer from every day. Australian supermarket should definitely learn from the plastic bag design here.
In terms of my project (Integrating Opal and Nimrod/K), I had one meeting with Ilkay, Maria and Jiang Wu and another meeting with Wilfred this week. Colin says my project needs much more than two month's work. The current version of Nimrod/K does not work with Kepler 2.X. Another student is updating Nimrod/K at Monash and it's going to take 4-6 weeks. In addition, the Opal actor cannot determine the order of files. It is not worthwhile to make Nimrod/K and Opal web service works in Kepler 1.0 and upgrade them to Kepler 2.X. Ilkay suggests that I should define a use case scenario and create a workflow in Kepler 2.X before I work on the integration part. Wilfred and Colin advise me to fix the Opal actor. I'm going to have a meeting with Ilkay and Wilfred to discuss my project plan further. I think fix the Opal actor, create a workflow and design an integration plan would be achievable while I'm at UCSD. The difficult part at the moment is to set up the development environment. Wai Keung I have spent hours on it but we made little progress. Certain part of the installation instructions on the Kepler web site is outdated. Apart from the installation, I need to get opal.xml workflow and the associated libraries.
Next week, we are going to have a cocktail party and promote MURPA program. We also plan to visit San Fransisco during the long weekends.
The weather at San Diego in winter is so pleasant, the apartment is fantastic and the people at UCSD are friendly. Everything at UCSD helps me recover very soon and I feel I'm in the right mode to work on my project. I particularly like the supermarket here. It's still open after midnight and it's got tons of food, much better than Australian supermarkets. The plastic bag design is brilliant. There is no need to rub the bag to open it. You pull the front bag and the next bag opens automatically. This design reminds me Don Normans's book "The design of everyday things". This book is the best introduction to the importance of usability in design: much of the value comes from the fact that it is not about computers but about all kinds of other things that we suffer from every day. Australian supermarket should definitely learn from the plastic bag design here.
In terms of my project (Integrating Opal and Nimrod/K), I had one meeting with Ilkay, Maria and Jiang Wu and another meeting with Wilfred this week. Colin says my project needs much more than two month's work. The current version of Nimrod/K does not work with Kepler 2.X. Another student is updating Nimrod/K at Monash and it's going to take 4-6 weeks. In addition, the Opal actor cannot determine the order of files. It is not worthwhile to make Nimrod/K and Opal web service works in Kepler 1.0 and upgrade them to Kepler 2.X. Ilkay suggests that I should define a use case scenario and create a workflow in Kepler 2.X before I work on the integration part. Wilfred and Colin advise me to fix the Opal actor. I'm going to have a meeting with Ilkay and Wilfred to discuss my project plan further. I think fix the Opal actor, create a workflow and design an integration plan would be achievable while I'm at UCSD. The difficult part at the moment is to set up the development environment. Wai Keung I have spent hours on it but we made little progress. Certain part of the installation instructions on the Kepler web site is outdated. Apart from the installation, I need to get opal.xml workflow and the associated libraries.
Next week, we are going to have a cocktail party and promote MURPA program. We also plan to visit San Fransisco during the long weekends.
Sunday, August 22, 2010
OLAP and data mining
Many senior managers and executives say they like businesss intelligence systems/EIS. But majority of data warehouse users still using Excels, reporting and data analysis tools, or their own customized applications to draw data from warehouses and transform it into business reports and charts. In general, these approaches work fine static data analysis with small amount of data.
Multidimensional databases and reporting system usually generate attractive sales presentations and demonstrations. Sales information can be viewed from various dimensions such as region, product type, time and sales person. OLAP enables users to explore enterprise data from different perspectives.
OLAP servers and desktop tools support high-speed analysis of data. Many verdors provide OLAP tools including Microsoft, SAP, Oracle and Cognos. Data manipulaiton in multidimensional databases can be very fast because they store the data in structures denormalized and optimized for speed. But multidimensional databases take huge amount of time to update. Software developers are attempting to deal with the update issue through the use of partitioning.
Data is valuable asset to organizations because it enables decision making. OLAP along with data mining, when incorporated into a data warehousing products, help decision makers analyze historical data and extract hidden patterns in data. OLAP provides drill down and roll up data analysis. Data mining tools enables supervised and unsupervised learning. Multidimensional analysis requires users to interact with the database to find information in the database. Data mining tool does not require users to specify a problem to be examined. For example a data mining tool in a supermarket database can find out which products customers usually buy together. Then, supermarket can provide special offers on these products or put them in adjacent shelves to generate more sales.
Multidimensional databases and reporting system usually generate attractive sales presentations and demonstrations. Sales information can be viewed from various dimensions such as region, product type, time and sales person. OLAP enables users to explore enterprise data from different perspectives.
OLAP servers and desktop tools support high-speed analysis of data. Many verdors provide OLAP tools including Microsoft, SAP, Oracle and Cognos. Data manipulaiton in multidimensional databases can be very fast because they store the data in structures denormalized and optimized for speed. But multidimensional databases take huge amount of time to update. Software developers are attempting to deal with the update issue through the use of partitioning.
Data is valuable asset to organizations because it enables decision making. OLAP along with data mining, when incorporated into a data warehousing products, help decision makers analyze historical data and extract hidden patterns in data. OLAP provides drill down and roll up data analysis. Data mining tools enables supervised and unsupervised learning. Multidimensional analysis requires users to interact with the database to find information in the database. Data mining tool does not require users to specify a problem to be examined. For example a data mining tool in a supermarket database can find out which products customers usually buy together. Then, supermarket can provide special offers on these products or put them in adjacent shelves to generate more sales.
Sunday, August 8, 2010
week 3 some thoughts after studying BI development cases
We have examined a few case studies of the development of business intelligence systems in week 3. Unlike any subjects in science which usually have refined theories, it can be concluded from the case studies that there is no ultimate approach to develop a BI system. Different firms use different methods to develop BI, including in – house development, outsourcing, adaptive development and many others which are not popular at the moment. The rationale behind companies’ choices is very complex and some companies even don’t have good reasons why they choose one approach rather than the other. As a student, I have been looking at all sorts of BI system development and hoping that I can extract some BI system development principles or rules from real cases to guide future practise. There are many similarities among technical details, but when I take social, cultural, people and external environment into consideration, it is extremely difficult to figure out what lead us to the successful BI development. Social paradigm suggests that we should look at a problem from different perspectives and try to understand each of them to improve overall understanding of the situation. But human brain can process limited amount of information at any given time. When I look at a case study from different angles, I don’t feel my understanding of the problem has significantly improved. Sometimes, it is even counterproductive that I feel the case is too complicated. We are living in an information age where information is at our fingertips. But how should we process and organize the information in an appropriate way so that we can process them in human brain? In the lecture, we have looked at case studies one by one. It seems that we have learned success and failures from each of them. But I want to know the connections between them so we can build knowledge to help us deal with a wider range of BI system development. In other words, knowledge specific to one company’s BI system does not have great value to other BI systems. I hope I can have more insight into the connections among a variety of BI systems at the end of the semester.
Sunday, August 1, 2010
week 2 data mining and BI application
I have changed one of my unit enrollments from distributed database to knowledge discovery and data mining this week. One reason is that distributed database unit has not changed its prescribed textbook for many years. It’s still using a book which was published in 1999. Technology keeps updating all the time, I don’t believe distributed database technology has not evolved during the last few years. Another important reason is that data mining has close relationship with business intelligence application. Studying these two units can give me more insight into data analysis within the context of business environment.
Business intelligence is information about a company's past performance that is used to help predict the company's future performance. Data mining allows users to sift through the enormous amount of information available in data warehouses, it is from this sifting process that business intelligence gems may be found. Data mining is intuitive, allowing for increased insight beyond data warehousing. An implementation of data mining in an organization will serve as a guide to uncovering unknown trends in historical data. It will also allow for statistical predictions, groupings and classifications of data. Data mining software allows users to analyze large databases to solve business decision-making problems. Data mining tools predict future trends and behaviors, allowing businesses to make proactive, knowledge-driven decisions. Data mining tools can answer business questions that traditionally were too time-consuming to resolve. As the semester progresses, it would be interesting to see how data mining technology is built into BI applications and how these two units support one another.
Business intelligence is information about a company's past performance that is used to help predict the company's future performance. Data mining allows users to sift through the enormous amount of information available in data warehouses, it is from this sifting process that business intelligence gems may be found. Data mining is intuitive, allowing for increased insight beyond data warehousing. An implementation of data mining in an organization will serve as a guide to uncovering unknown trends in historical data. It will also allow for statistical predictions, groupings and classifications of data. Data mining software allows users to analyze large databases to solve business decision-making problems. Data mining tools predict future trends and behaviors, allowing businesses to make proactive, knowledge-driven decisions. Data mining tools can answer business questions that traditionally were too time-consuming to resolve. As the semester progresses, it would be interesting to see how data mining technology is built into BI applications and how these two units support one another.
week1 introduction to BI application
Business Intelligence application is a fascinating unit. Rob gave a broad description on BI in week one lecture, but there were things I don’t agree with. Rob said some BI applications were powerful enough to be built directly on top of the existing systems. It would be fantastic if we could plug and play BI just like how we use USB drive, but there are issues around BI which need to be addressed if we want to reach the full potential of BI. Many organizations today have dozens of different information systems and each of those systems may be built with different languages and data format. Those systems usually have multiple version of the same fact and don’t talk with each other. In other words, data integrity and consistency is not guaranteed. BI application draws data from a wide range of information system to support decision making. Garbage - in and garbage – out theory suggests that the quality of information produced by computer applications is only as good as the data being used. If the input data is poor, no matter how good a BI application is, it will produce wrong information which has negative value to decision makers. So before a company implements BI, it must cleanse the data and integrate information systems. This can be done through implementing an ERP system in which a centralized database system and web services are used to extract and manage data from multiple system.
I didn’t ask Rob why BI can even be built on top of existing system directly. But I guess it’s probably because many vendors lie about what their BI applications can do to generate more sales.
I didn’t ask Rob why BI can even be built on top of existing system directly. But I guess it’s probably because many vendors lie about what their BI applications can do to generate more sales.
Sunday, April 27, 2008
Week 6 Use Case Diagram
This week we were learning use case modeling. It is an objected-oritented approach to model user requirements. The former diagram is a use case diagram created to illustrarte the behavior of shoes stock management system.
If we have drawn an enent table, it is easy to draw a use case diagram. But system analysists have to pretend that it is difficult to draw it. The users are going to pay money. We must prove our work is hard and valuable.
VP is a very useful tool to draw diagram. But it is complex for beginner to use it. I have never seen some of the functions it provides.
Wednesday, April 16, 2008
week 5 traditional approach and developing event table
In week 5, we were talking about SDLC in traditional approach. The data flow diagram is vital in the traditional approach. But it seems that system analysts do not like it. The main reason is DFD view the business process as a set and mechanistic hierarchy. Today, more and more business are becoming dynamic so that DFD can not define all business well.
I have tried to draw an level-0 DFD for an online sales system.
I also discussed event tables with my classmates. I found that most people were trying to add lots of events into the event table. Somebody's event table has more than 40 rows! I think it is good to view a system from different angles. But they are too ambicious. We are beginners and we are going to draw user case diagram and write user case narratives. It is just too complex if we add every thing into one system. Thus, I modified my event table and followed "CRUD". It is a very logic way to include all important events although it is not very flexiable.
Sunday, April 6, 2008
week 4 mock interview and event table
In week4 , we had an interview with tutor. An draft event table is drawn here, the red words are unsure part of the event table, I am trying to improve the quality of event table.
Thursday, March 27, 2008
Week3 requirement gathering and preparing interview
In week 3, we have stepped into analysis phase of SDLC and mainly talked about gathering information. We were also required to listen to an audio file and prepare interview questions about shoes stock management in a shoes store.
Requirement gathering comprises dertermining requirement, gathering information and validating requirement. At first galance, it seems that gathering requirement is an easy work, but it turns out to be fairly difficult and need significant effort. A system analyst cannot gather requirement well until he possesses of sufficient technical skills, comprehensive understading of business process and good communication skills. Peter said business people do not like IT people. If they do not trust you, they will not tell you the right things. I think peter's words address one aspect of communication issues. Peter also talked about his own experience about building a system for a stationary shop in a school. There was a problem with the sales database. Every sale was recorder twice. Peter checked the programe code and everything else but still cound not find the problem. Then he went to the shop and watch how people scan item and he found that people scanned items twice! This example imply that watch how people doing things is a effective way to figure out user's requirement.
We also started assignment 1 in week3. The assignment is buliding a shoes stock management system for a shoes shop. We were required to listen to an audio file and prepare questions for a mock interview. Assignment1a is mainly drawing an event table. After I listened to the audio, I found that the system required for stock management is simple. The owner of the shop only need some kind of system to manage inventory, deal with returned shoes and perform sales processing functions. For instance, zac wants to know the location and availability of each particular shoes and the system can help him record these imformation.
Requirement gathering comprises dertermining requirement, gathering information and validating requirement. At first galance, it seems that gathering requirement is an easy work, but it turns out to be fairly difficult and need significant effort. A system analyst cannot gather requirement well until he possesses of sufficient technical skills, comprehensive understading of business process and good communication skills. Peter said business people do not like IT people. If they do not trust you, they will not tell you the right things. I think peter's words address one aspect of communication issues. Peter also talked about his own experience about building a system for a stationary shop in a school. There was a problem with the sales database. Every sale was recorder twice. Peter checked the programe code and everything else but still cound not find the problem. Then he went to the shop and watch how people scan item and he found that people scanned items twice! This example imply that watch how people doing things is a effective way to figure out user's requirement.
We also started assignment 1 in week3. The assignment is buliding a shoes stock management system for a shoes shop. We were required to listen to an audio file and prepare questions for a mock interview. Assignment1a is mainly drawing an event table. After I listened to the audio, I found that the system required for stock management is simple. The owner of the shop only need some kind of system to manage inventory, deal with returned shoes and perform sales processing functions. For instance, zac wants to know the location and availability of each particular shoes and the system can help him record these imformation.
Wednesday, March 5, 2008
FIT2001 Week1&2 Understanding the nature of system analysis
As the title indicated, the goal of the first two weeks' exercise is to help student understanding the nature of system analysis. During the lecture, I have studied system analysis theoratically. The more theoratical, the more difficult to understand the nature of system analysis. However,if students can combine the theory with real world, we probably can have a better understanding. I think that's why the exercise of the tutorial is requiring students to analysis water supply system and food system. students were split into several groups and each group discuss about the two systems.
Everyone is familiar with the water supply system and the food system because they have significant impact on our daily life. But the question still remains: Do we really understand the water supply system and food system? First of all, what we have seen everyday is small part of the systems, a large amount of elements are functioning and interactiong behind our sight. In additional, the quality of our life has improved dramatically during the last a few decades. People are more concerned about the higher level activities rather then the basic activities. I have clicked the web link on the tutorial notes and I found an very interesting story. A mum asked her son, "where does milk come from?" The boy responded loudly, "From supermarkets!" This story is an extreme example to point out people's ignorance of basic systems. Furthermore, although water system and food system seem to be simple at first galance, they are becoming more and more complex along with the development of our society.
When I discussed with my gruop members, I noticed that we were leraning the water and food system in the same way like blind people learning an elephant. Everyone of us had different view of the systems and we were trying to construct the systems in our mind by combining everyone's view together. We also did lots of research on websites. However, it was absolutely a mass, we finally got lost in the systems. I think it is mainly because we were studying complex systems in an arbitary way, we were strongly depending on our instinct. We do need an effective and efficient approach to study system analysis.
As communication skill is vital for system analysists, our homework was to perpare an presentation for water supply and food system. My job was to perpare a presentation for food system in the view point of standard authorities. In order to conquer the difficulties I mentioned previously, I decided to study the food system succintly. People may ask me how can you understan a complex system succinctly? Well, my answer is we are not trying to build a real food system. I am a beginner, I can identitfy the key elements of food system and describe these key elements in detail. I compared a food standard authority in a small town with the food authority in China. I quickly identified setting standards and executing standards are the key elements for any food autority. Then, I could foucus my mind on these two single elements. During the presentation, I still kept the principle of succinctness. For instince, when I was talking about setting standards for farmers, I didn't tell everyone a food authority should make blah.....blah...blah... standards for farmers. It is very boring and inefficient! Alternatively, I told them a story :I always buy some lunchbox size apples in supermarkets and I found that the quality of the apples varies a great deal in different markets. " What is the implication?" I encouraged everyone to think. Then I told everyone it mainly because farms fertilized and irritated the same kind of apples in different ways. In order to ensure that the food in the market is of satisfactory quality, food authority has to make standards for fertilization, irritation, solid and so on.
I want to stop writing in here. I don't want to make it a long article. The last thing I would like to say is what I am going to write next week. When I finish reading the first few chapters of the textbook, I am going to write a summary or argument to refine my thinking. Because Newton's story taught me when I read giant's book, I must try to stand on their shoulder so that I can look further than the giant!
Everyone is familiar with the water supply system and the food system because they have significant impact on our daily life. But the question still remains: Do we really understand the water supply system and food system? First of all, what we have seen everyday is small part of the systems, a large amount of elements are functioning and interactiong behind our sight. In additional, the quality of our life has improved dramatically during the last a few decades. People are more concerned about the higher level activities rather then the basic activities. I have clicked the web link on the tutorial notes and I found an very interesting story. A mum asked her son, "where does milk come from?" The boy responded loudly, "From supermarkets!" This story is an extreme example to point out people's ignorance of basic systems. Furthermore, although water system and food system seem to be simple at first galance, they are becoming more and more complex along with the development of our society.
When I discussed with my gruop members, I noticed that we were leraning the water and food system in the same way like blind people learning an elephant. Everyone of us had different view of the systems and we were trying to construct the systems in our mind by combining everyone's view together. We also did lots of research on websites. However, it was absolutely a mass, we finally got lost in the systems. I think it is mainly because we were studying complex systems in an arbitary way, we were strongly depending on our instinct. We do need an effective and efficient approach to study system analysis.
As communication skill is vital for system analysists, our homework was to perpare an presentation for water supply and food system. My job was to perpare a presentation for food system in the view point of standard authorities. In order to conquer the difficulties I mentioned previously, I decided to study the food system succintly. People may ask me how can you understan a complex system succinctly? Well, my answer is we are not trying to build a real food system. I am a beginner, I can identitfy the key elements of food system and describe these key elements in detail. I compared a food standard authority in a small town with the food authority in China. I quickly identified setting standards and executing standards are the key elements for any food autority. Then, I could foucus my mind on these two single elements. During the presentation, I still kept the principle of succinctness. For instince, when I was talking about setting standards for farmers, I didn't tell everyone a food authority should make blah.....blah...blah... standards for farmers. It is very boring and inefficient! Alternatively, I told them a story :I always buy some lunchbox size apples in supermarkets and I found that the quality of the apples varies a great deal in different markets. " What is the implication?" I encouraged everyone to think. Then I told everyone it mainly because farms fertilized and irritated the same kind of apples in different ways. In order to ensure that the food in the market is of satisfactory quality, food authority has to make standards for fertilization, irritation, solid and so on.
I want to stop writing in here. I don't want to make it a long article. The last thing I would like to say is what I am going to write next week. When I finish reading the first few chapters of the textbook, I am going to write a summary or argument to refine my thinking. Because Newton's story taught me when I read giant's book, I must try to stand on their shoulder so that I can look further than the giant!
Subscribe to:
Posts (Atom)
