codetoad.com
  ASP Shopping CartForum & BBS
  - all for $20 from CodeToad Plus!
  
  Home || ASP | ASP.Net | C++/C# | DHTML | HTML | Java | Javascript | Perl | VB | XML || CodeToad Plus! || Forums || RAM 
Search Site:



Home » ASP » Article

Internal Search Engine

Article by: Stevan Rodrigues (2/4/2004)
Bookmark us now! Add to Favourites
Email a friend!Tell a friend
Summary: The Search Engine module will search an entire page also dynamic pages for matching keyword(s) or a phrase and will count how many times the keyword(s) or phrase are found on the page, and display the results with the highest matches first. The module will search all files with the extensions that you can easily place the extension name into the web.config file where indicated. Files or folders that you don't want searched can be placed in the web.config where indicated, so these files and folders are not searched.
Viewed: 46805 times Rating (7 votes): 
 4.3 out of 5
 Rate this Article  Read Comments  Post Comments

Internal Search Engine




Sample Image


Introduction


The Search Engine module will search an entire page also dynamic pages
for matching keyword(s) or a phrase and will count how many times the keyword(s)
or phrase are found on the page, and display the results with the highest matches
first. The module will search all files with the extensions that you can easily
place the extension name into the web.config file where indicated. Files
or folders that you don't want searched can be placed in the web.config
where indicated, so these files and folders are not searched.


Note: It is best suited for small sites. Also you can modify this code to crawl
pages internally by using regular expressions. For larger site you will need
to write to XML file once periodically and then read from XML file.


Background


Site Search Engine helps a user to trace the pages of his interest. When I was working on an ASP.NET project, I had to add the site search module. I had one in ASP but not in .NET. Hence the birth of this site search engine. My first version was just a single web form and I had not exploited the full features of the object oriented .NET language. In my spare time, I reworked my code to make the maximum use of the object oriented language. For this article, I further enhanced my design on the basis of the experience and the good practices suggested by different authors.


Source Code Overview


The structure of the Site Search Engine is as follows:



Classes


The ability to define a class and create instances of classes is one of the most important capabilities of any object-oriented language. In the coming section, we see the classes that we have used in the search module.

























Class Name Description
SiteSearch Class for the web form where the user can search the site for certain words.
Searches.CleanHtml Class to clean the HTMl content
Searches.Page Class to store data of the pages
Searches.PagesDataset Class to create and store results in dataset
Searches.Site Class to read the configurations of the site
Searches.UserSearch Class to store the search information per user

SiteSearch.aspx


Web Forms are one of the new, exciting features in Microsoft's .NET initiative. SiteSearch.aspx is a web form which is also the start page for the search module.


A Web Forms page consists of a page (ASPX file) and a code behind file (.aspx.cs file or .aspx.vb file). Our web form consists of SiteSearch.aspx and SiteSearch.aspx.vb. I will be treating them simultaneously touching on the main elements of the web form.


ASP.NET is an event-driven programming environment. We will see some event handlers and methods in the coming section.


Page_Load


The server controls are loaded on the Page object and the view state information is available at this point. So this is where we put the code to change control settings or display text on the page. Here we check if the search contains text and then call the SearchSite method. The DataGrid content is set by calling the BindDataGrid method. ViewState("SortExpression") is used to store the sort expression.


      '***********************************************
'
' Page_Load event
'
' Add code to this event.
'
'***********************************************
Private Sub Page_Load(ByVal sender As System.Object, _
ByVal e As System.EventArgs) Handles MyBase.Load
Dim strSearchWords As String
'If there is no words entered by the user to search
' for then dont carryout the file search routine
pnlSearchResults.Visible = False
strSearchWords = Trim(Request.Params("search"))

If Not strSearchWords.Equals("") Then
Searchs.Site.ApplicationPath = String.Format("http://{0}{1}", Request.ServerVariables("HTTP_HOST"), Request.ApplicationPath)
sSite = SearchSite(strSearchWords)

If Not IsNothing(sSite.PageDataset) Then
pnlSearchResults.Visible = True
lblSearchWords.Text = sSite.SearchWords

If ViewState("SortExpression") Is Nothing Then
ViewState("SortExpression") = "MatchCount Desc"
End If

BindDataGrid(ViewState("SortExpression"))
lblTotalFiles.Text = sSite.TotalFilesSearched
lblFilesFound.Text = sSite.TotalFilesFound
End If
End If
End Sub

Search


The main call to the search takes place in this method. UserSearch class which we will cover shortly stores the entire search information and the result of the search. UserSearch object, i.e. srchSite is created and its properties like SearchWords and SearchCriteria assigned. Also srchSite.Search method is called.

      '************************************************************
'
' SearchSite method
'
' The sSite.PageDataset is used to populate the datagrid.
'
'************************************************************
Private Function SearchSite(ByVal strSearch_
As String) As Searchs.UserSearch
Dim srchSite As Searchs.UserSearch
srchSite = New Searchs.UserSearch()
'Read in all the search words into one variable
srchSite.SearchWords = strSearch

If Phrase.Checked Then
srchSite.SearchCriteria = Searchs.SearchCriteria.Phrase
ElseIf AllWords.Checked Then
srchSite.SearchCriteria = Searchs.SearchCriteria.AllWords
ElseIf AnyWords.Checked Then
srchSite.SearchCriteria = Searchs.SearchCriteria.AnyWords
End If

srchSite.Search(Server.MapPath("./"))
Return srchSite
End Function

DataGrid


The DataGrid control renders a multi-column, fully templated grid and is by far the most versatile of all data bound controls. Moreover the DataGrid control is the ASP.NET control of choice for data reporting. Hence, I have used it to display the search results. Since the focus of the article is the internal search engine, I will just give a brief overview of the features of the DataGrid used here.


Databinding

Data binding is the process of retrieving data from a source and dynamically associating it to a property of a visual element. Because a DataGrid handles (or at least has in memory) more items simultaneously, you should associate the DataGrid explicitly with a collection of data—that is, the data source.


The content of a DataGrid is set by using its DataSource property. The entire search result is stored in the sSite.PageDataset.Tables("Pages"). Hence the content of the DataGrid is set to dvwPages i.e. sSite.PageDataset.Tables("Pages").DefaultView. BindDataGrid method is called every time the page loads.

      '************************************************************
'
' BindDataGrid method
'
' The sSite.PageDataset is used to populate the datagrid.
'
'************************************************************
Private Sub BindDataGrid(ByVal strSortField As String)
Dim dvwPages As DataView
dvwPages = sSite.PageDataset.Tables("Pages").DefaultView
dvwPages.Sort = strSortField
dgrdPages.DataSource = dvwPages
dgrdPages.DataBind()
End Sub

The control has the ability to automatically generate columns that are based on the structure of the data source. Auto-generation is the default behavior of DataGrid, but you can manipulate that behavior by using a Boolean property named AutoGenerateColumns. Set the property to False when you want the control to display only the columns you explicitly add to the Columns collection. Set it to True (the default) when you want the control to add as many columns as is required by the data source. Auto-generation does not let you specify the header text, nor does it provide text formatting. Hence, here I have set it to False. You typically bind columns using the <columns> tag in the body of the <asp:datagrid> server control.

 <Columns>
<asp:TemplateColumn>
<ItemTemplate>
<%# DisplayTitle(Container.DataItem( "Title" ), _
Container.DataItem( "Path" )) %>
<br>
<%# Container.DataItem( "Description" ) %>
<br>
<span class="Path">
<%# DisplayPath(Container.DataItem( "Path" )) %>
</span>
<br>
<br>
</ItemTemplate>
</asp:TemplateColumn>
</Columns>

DisplayTitle method and DisplayPath method are used to display customized information to the columns in the DataGrid.

      '****************************************
'
' DisplayTitle method
'
' Display title of searched pages
'
'****************************************
Protected Function DisplayTitle(ByVal Title _
As String, ByVal Path As String) As String
Return String.Format("<A href="{1}">{0}</A>", Title, Path)
End Function

'****************************************
'
' DisplayPath method
'
' Path of the file is returned
'
'****************************************
Protected Function DisplayPath(ByVal Path As String) As String
Return String.Format("{0}{1}/{2}", _
Request.ServerVariables("HTTP_HOST"), _
Request.ApplicationPath, Path)
End Function

Pagination

Unlike the DataList control, the DataGrid control supports data pagination, that is, the ability to divide the displayed data source rows into pages. The size of our data source easily exceeds the page real estate. So to preserve scalability on the server and to provide a more accessible page to the user, you display only a few rows at a time. To enable pagination of the DataGrid control, you need to tell the control about it. You do this through the AllowPaging property.


The pager bar is an interesting and complimentary feature offered by the DataGrid control to let users easily move from page to page. The pager bar is a row displayed at the bottom of the DataGrid control that contains links to available pages. When you click any of these links, the control automatically fires the PageIndexChanged event and updates the page index accordingly. dgrdPages_PageIndexChanged is called when the page index changes.

      '*****************************************************************
'
' dgrdPages_PageIndexChanged event
'
' The CurrentPageIndex is Assigned the page index value.
' The datagrid is then populated using the BindDataGrid function.
'
'*****************************************************************
Protected Sub dgrdPages_PageIndexChanged(ByVal s As Object, _
ByVal e As DataGridPageChangedEventArgs) _
Handles dgrdPages.PageIndexChanged
dgrdPages.CurrentPageIndex = e.NewPageIndex
BindDataGrid(ViewState("SortExpression"))
End Sub

You control the pager bar by using the PagerStyle property’s Mode attribute. Values for the Mode attribute come from the PagerMode enumeration. Here we have chosen a detailed series of numeric buttons, each of which points to a particular page.

<PagerStyle CssClass="GridPager" Mode="NumericPages"></PagerStyle>

Sorting

The DataGrid control does not actually sort rows, but it provides good support for sorting as long as the sorting capabilities of the underlying data source are adequate. The data source is always responsible for returning a sorted set of records based on the sort expression selected by the user through the DataGrid control’s user interface. The built-in sorting mechanism is triggered by setting the AllowSorting property to True.


dgrdPages_SortCommand is called to sort the DataGrid. The SortCommand event handler knows about the sort expression through the SortExpression property, which is provided by the DataGridSortCommandEventArgs class. In our code, the sort information is persisted because it is stored in a slot in the page’s ViewState collection.


Note: In my pages, I have disabled the header but if the header is shown, you can use it to sort the DataGrid.

      '*****************************************************************
'
' dgrdAdditionalItems_SortCommand event
'
' The ViewState( "SortExpression" ) is Assigned
' the sort expression value.
' The datagrid is then populated using the BindDataGrid function.
'
'*****************************************************************
Protected Sub dgrdPages_SortCommand(ByVal s As Object, _
ByVal e As DataGridSortCommandEventArgs) _
Handles dgrdPages.SortCommand
ViewState("SortExpression") = e.SortExpression
BindDataGrid(ViewState("SortExpression"))
End Sub

Page.vb



The role of the Page object is to store the data related to each page of the site.


The Page class defines the following properties





















Path Stores the path of the file
Title Stores the text in HTML title tag
Keywords Stores the text in HTML meta keywords tags
Description Stores the text in HTML meta description tags
Contents Stores the text in HTML page
Matchcount Stores the matches found in HTML page
        '***************************************************
'
' Path Property
'
' Assign and retreive path of the file
'
'***********************************************

Public Property Path() As String
Get
Return m_path
End Get
Set(ByVal Value As String)
m_path = Value
End Set
End Property

'***********************************************
'
' Title Property
'
'Assign and retreive title of the file
'
'***********************************************
Public Property Title() As String
Get
Return m_title
End Get
Set(ByVal Value As String)
m_title = Value
End Set
End Property

'***********************************************
'
' Keywords Property
'
' Assign and retreive Keywords
' (meta tags) of the file
'
'***********************************************
Public Property Keywords() As String
Get
Return m_keywords
End Get
Set(ByVal Value As String)
m_keywords = Value
End Set
End Property


'***********************************************
'
' Description Property
'
' Assign and retreive description
' (meta tags) of the file
'
'***********************************************
Public Property Description() As String
Get
Return m_description
End Get
Set(ByVal Value As String)
m_description = Value
End Set
End Property

'***********************************************
'
' Contents Property
'
' Assign and retreive contents of the file
'
'***********************************************
Public Property Contents() As String
Get
Return m_contents
End Get
Set(ByVal Value As String)
m_contents = Value
End Set
End Property


'***********************************************
'
' Contents Property
'
' Assign and retreive MatchCount of the file
'
'***********************************************
Public Property MatchCount() As Integer
Get
Return m_matchcount
End Get
Set(ByVal Value As Integer)
m_matchcount = Value
End Set
End Property

The Page class has two private methods and two public methods. It defines the following methods:


CheckFileInfo Method


It is a public method which checks if title, description and content exists. If the text for title is empty then it assigns the default value "No Title". If the text for description is empty then it either assigns the contents of the file or default value "There is no description available for this page".

      '*************************************************
'
' CheckFileInfo method
'
' Subroutine to the check the file contains
' title and decription
'
'*************************************************
Public Sub CheckFileInfo()

'If the page contains no title then Page Title
' variable the appropriate message to display
If IsNothing(m_title) Or m_title.Trim().Equals("") Then
m_title = "No Title"
End If


'If the page contains no title then Page Description
'variable the appropriate message to display
If IsNothing(m_description) Or m_description.Trim().Equals("") Then
If IsNothing(m_contents) Or m_contents.Trim().Equals("") Then
m_description = _
"There is no description available for this page"
Else
If m_contents.Length > 200 Then
m_description = m_contents.Substring(0, 200)
Else
m_description = m_contents
End If
End If
End If
End Sub

Search method


Search method is a public method which calls SearchPhrase and SearchWords methods depending on the search criteria. SearchPhrase method searches for phrases while SearchWords searches for all or any words. Both these methods use regular expressions to search the files.

            '*******************************************
'
' Search method
'
' Subroutine to the search file
'
'*******************************************
Public Sub Search(ByVal strSearchWords As String, _
ByVal SrchCriteria As SearchCriteria)

'If the user has choosen to search by phrase
If SrchCriteria = SearchCriteria.Phrase Then
SearchPhrase(strSearchWords)
'Else the search is either by all or any words
Else
SearchWords(strSearchWords, SrchCriteria)
End If

End Sub

'*******************************************
'
' SearchPhrase method
'
' Subroutine to the search file
'
'*******************************************
Private Sub SearchPhrase(ByVal strSearchWords As String)
Dim regexp As Regex
Dim mtches As MatchCollection

'Set the pattern to search for
regexp = New Regex("", RegexOptions.IgnoreCase)

'Search the file for the phrase
mtches = regexp.Matches(m_contents, _
String.Format("\b{0}\b", strSearchWords), _
RegexOptions.IgnoreCase)

'Check to see if the phrase has been found
If mtches.Count > 0 Then
'Get the number of times the phrase is matched
m_matchcount = mtches.Count
End If
End Sub

'*******************************************
'
' SearchWords method
'
' Subroutine to the search file
'
'*******************************************
Private Sub SearchWords(ByVal strSearchWords _
As String, ByVal SrchCriteria As SearchCriteria)
Dim regexp As Regex
Dim intSearchLoopCounter As Integer
Dim sarySearchWord As String()
'Array to hold the words to be searched for
Dim mtches As MatchCollection

'Split each word to be searched up and place in an array
sarySearchWord = Split(Trim(strSearchWords), " ")

regexp = New Regex("", RegexOptions.IgnoreCase)

'Loop round to search for each word to be searched
For intSearchLoopCounter = 0 To UBound(sarySearchWord)

'Set the pattern to search for
mtches = regexp.Matches(m_contents, _
String.Format("\b{0}\b", _
sarySearchWord(intSearchLoopCounter)),_
RegexOptions.IgnoreCase)

If SrchCriteria = SearchCriteria.AnyWords Then
m_matchcount = m_matchcount + mtches.Count
Else
'Check to see if any of the words have been found
If mtches.Count > 0 Then
'Get the number of times the search word is matched
If m_matchcount = 0 Or _
(m_matchcount > 0 And _
m_matchcount > mtches.Count) Then
m_matchcount = mtches.Count
End If
Else
'If the search word is not found then
'set the search found variable back to false
'as one of the words has not been found
m_matchcount = 0
Exit Sub

End If
End If
Next
End Sub

UserSearch.vb



It contains the following properties:















SearchCriteria The user choice of search is stored and retrieved from here
SearchWords The search words used by the user is stored and retrieved from here
TotalFilesSearched Total Files Searched is read from here
TotalFilesFound Total Files Searched found is read from here
        '**********************************************************
'
' SearchCriteria Property
'
' Assign and retrieve SearchCriteria of the site
'
'**********************************************************
Public Property SearchCriteria() As Searchs.SearchCriteria
Get
Return m_searchCriteria
End Get
Set(ByVal Value As Searchs.SearchCriteria)
m_searchCriteria = Value
End Set
End Property

'**********************************************************
'
' SearchWords Property
'
'Assign and retrieve SearchWords of the site
'
'**********************************************************
Public Property SearchWords() As String
Get
Return m_searchWords
End Get
Set(ByVal Value As String)
m_searchWords = Value
End Set
End Property

'**********************************************************
'
' TotalFilesSearched Property
'
' Retrieve TotalFilesSearched of the site
'
'**********************************************************
Public ReadOnly Property TotalFilesSearched() As Integer
Get
Return m_totalFilesSearched
End Get
End Property

'**********************************************************
'
' TotalFilesFound Property
'
' Retrieve TotalFilesFound of the site
'
'**********************************************************
Public ReadOnly Property TotalFilesFound() As Integer
Get
Return m_totalFilesFound
End Get
End Property

'**********************************************************
'
' PageDataset Shared Property
'
' Retrieve data of the entire site of the site
'
'**********************************************************
Public ReadOnly Property PageDataset() As DataSet
Get
Return m_dstPages
End Get
End Property

Search Method


Actual processing of the search begins here. DataSet to store the search results is created here and ProcessDirectory method is called.

        '********************************************
'
' Search Method
'
' Search the entire site
'
'********************************************
Public Function Search(ByVal targetDirectory As String) As DataSet
'If the site is in English then use the server HTML encode method
If Searchs.Site.EnglishLanguage = True Then
'Replace any HTML tags with the HTML codes
'for the same characters (stops people entering HTML tags)
m_searchWords = m_page.Server.HtmlEncode(m_searchWords)
'If the site is not english just change the script tags
Else
'Just replace the script tag <> with HTML encoded < and >
m_searchWords = Replace(m_searchWords, "<", "<", 1, -1, 1)
m_searchWords = Replace(m_searchWords, ">", ">", 1, -1, 1)
End If
If m_dstPages Is Nothing Then
m_dstPages = Searchs.PagesDataset.Create()
End If
ProcessDirectory(targetDirectory)
Return m_dstPages
End Function

ProcessDirectory Method


The ProcessDirectory loops through all the files and calls the ProcessFile method. Later, it also loops through the subdirectories and calls itself.

       '*********************************************
'
' ProcessDirectory Method
'
' Files in the directories are searched
'
'********************************************
Private Sub ProcessDirectory(ByVal targetDirectory As String)
Dim fileEntries As String()
Dim subdirectoryEntries As String()
Dim filePath As String
Dim subdirectory As String

fileEntries = Directory.GetFiles(targetDirectory)

' Process the list of files found in the directory

For Each filePath In fileEntries
m_totalFilesSearched += 1
ProcessFile(filePath)
Next filePath

subdirectoryEntries = Directory.GetDirectories(targetDirectory)
' Recurse into subdirectories of this directory

For Each subdirectory In subdirectoryEntries

'Check to make sure the folder about to be searched
'is not a barred folder if it is then don't search
If Not InStr(1, Searchs.Site.BarredFolders, _
Path.GetFileName(subdirectory), vbTextCompare) > 0 Then
'Call the search sub prcedure to search the web site
ProcessDirectory(subdirectory)
End If

Next subdirectory

End Sub 'ProcessDirectory

ProcessFile Method


The ProcessFile calls the GetInfo which returns the Searchs.Page object which contains all the information of the particular file. Later, it checks if the matchcount is greater than 0 and calls the CheckFileInfo to clean up the information stored in the Page object. It then stores the file in the PagesDataset.

        '*******************************************************
'
' ProcessFile Method
'
' Real logic for processing found files would go here.
'
'*******************************************************
Private Sub ProcessFile(ByVal FPath As String)
Dim srchFile As Searchs.Page

srchFile = GetInfo(FPath)
If Not IsNothing(srchFile) Then

srchFile.Search(m_searchWords, m_searchCriteria)
If srchFile.MatchCount > 0 Then
m_totalFilesFound += 1
'Response.Write(srchFile.Contents)
srchFile.CheckFileInfo()
Searchs.PagesDataset.StoreFile(m_dstPages, srchFile)
End If

End If

End Sub 'ProcessFile

GetInfo Method


The GetInfo method's main task is to get the data of the file. It calls the GetFileInfo where much of the work is done.

        '*****************************************************************
'
' GetInfo Method
'
' File data is picked in this method
'
'*****************************************************************
Private Function GetInfo(ByVal FPath As String) As Searchs.Page

Dim fileInform As New FileInfo(FPath)
Dim sr As StreamReader
Dim srchFile As New Searchs.Page()
Dim strBldFile As New StringBuilder()
Dim strFileURL As String 'Holds the path to the file on the site


'Check the file extension to make sure the file
'is of the extension type to be searched

If InStr(1, Searchs.Site.FilesTypesToSearch, _
fileInform.Extension, vbTextCompare) > 0 Then
'm_page.Trace.Warn("File ext.", fileInform.Extension)
'Check to make sure the file about to be searched
'is not a barred file if it is don't search the file
If Not InStr(1, Searchs.Site.BarredFiles, _
Path.GetFileName(FPath), vbTextCompare) > 0 Then
'm_page.Trace.Warn("File", FPath)

If Not File.Exists(FPath) Then
'm_page.Trace.Warn("Error", _
'String.Format("{0} does not exist.", FPath))
'Add throw excetion here
'
'
Return Nothing
End If

GetFileInfo(FPath, srchFile)

Return srchFile

End If
End If
Return Nothing
End Function

GetFilePath Method


The GetFilePath method coverts local folder path to reflect the URL of the site.

        '*****************************************
'
' GetFilePath Method
'
' File path is modfied to be displayed
' as hyperlink in this method
'
'*****************************************
Private Sub GetFilePath(ByVal strFileURL As String,_
ByVal srchFile As Searchs.Page)
'Turn the server path to the file into a URL path to the file
strFileURL = Replace(strFileURL, m_page.Server.MapPath("./"), "")

'Replace the NT backslash with the internet
'forward slash in the URL to the file

strFileURL = Replace(strFileURL, "\", "/")

'Encode the file name and path into the URL code method

strFileURL = m_page.Server.UrlEncode(strFileURL)

'Just incase it's encoded any backslashes
strFileURL = Replace(strFileURL.Trim(), _
"%2f", "/", vbTextCompare)
srchFile.Path = strFileURL

m_page.Trace.Warn("Url", srchFile.Path)
End Sub

GetFileInfo Method


Here the chunk of the data in the page is retrieved. The file content is either
read from the source if the files are static using the GetStaticFileContent
method. If the files are dynamic then content is retreived from server using
the GetDynamicFileContent method. Title information is retrieved
from the title tags, and description and keywords from meta tags by calling
the GetMetaContent method. Contents of the file is stripped from
the HTML page by calling Searchs.CleanHtml.Clean method.


        '**********************************************
'
' GetFileInfo Method
'
' File data is picked in this method
'
'**********************************************
Private Sub GetFileInfo(ByVal FPath As String, _
ByVal srchFile As Searchs.Page)
Dim fileInform As New FileInfo(FPath)
Dim strBldFile As New StringBuilder()

GetFilePath(FPath, srchFile)

If InStr(1, Searchs.Site.DynamicFilesTypesToSearch, fileInform.Extension, vbTextCompare) > 0 Then
m_page.Trace.Warn("Path", String.Format("{0}/{1}", "", srchFile.Path))
GetDynamicFileContent(srchFile)
Else
GetStaticFileContent(FPath, srchFile)
End If

If Not srchFile.Contents.Equals("") Then


srchFile.Contents = sr.ReadToEnd()

'Read in the title of the file
srchFile.Title = GetMetaContent(srchFile.Contents,_
"<title>", "</title>")
'm_page.Trace.Warn("Page Title", strPageTitle)

'Read in the description meta tag of the file
srchFile.Description = GetMetaContent(srchFile.Contents,_
"<meta name=""description"" content=""", ",""">")
'm_page.Trace.Warn("
Page Desc", strPageDescription)

'Read in the keywords of the file
srchFile.Keywords = GetMetaContent(srchFile.Contents,_
"
<meta name=""keywords"" content=""", ",""">")
'm_page.Trace.Warn("Page Keywords", strPageKeywords)

srchFile.Contents = _
Searchs.CleanHtml.Clean(srchFile.Contents)

srchFile.Contents = _
strBldFile.AppendFormat("{0} {1} {2} {3}", _
srchFile.Contents, srchFile.Description, _
srchFile.Keywords, srchFile.Title).ToString.Trim()
'm_page.Trace.Warn("File Info", strBldFile.ToString)
<span class='vb-keyword'>End</span> <span class='vb-keyword'>If</span>End Sub

'*********************************************************************
'
' GetStaticFileContent Method
'
' File Content is picked in this method
'
'*********************************************************************
Private Sub GetStaticFileContent(ByVal FPath ByVal String, ByVal srchFile As Searchs.Page)
Dim sr As StreamReader

sr = File.OpenText(FPath)

Try
srchFile.Contents = sr.ReadToEnd()
sr.Close()
Catch ex As Exception
'm_page.Trace.Warn("Error", ex.Message)
End Try
End Sub

'*********************************************************************
'
' GetDynamicFileContent Method
'
' File Content is picked in this method
'
'*********************************************************************
Private Sub GetDynamicFileContent(ByVal srchFile As Searchs.Page)
Dim wcMicrosoft As System.Net.WebClient
Dim objUTF8Encoding As UTF8Encoding
Dim strResponse As String

wcMicrosoft = New System.Net.WebClient()
objUTF8Encoding = New UTF8Encoding()
srchFile.Contents = objUTF8Encoding.GetString( _
wcMicrosoft.DownloadData(String.Format("{0}/{1}", Searchs.Site.ApplicationPath, srchFile.Path)))

End Sub

GetMetaContent Method


GetMetaContent method uses regular expressions to strip the tags and get the required information.

        '************************************************
'
' GetMetaContent Method
'
' Metacontent is stripped in this method
'
'************************************************
Private Function GetMetaContent(ByVal strFile As String, _
ByVal strMetaStart As String, ByVal strMetaEnd As String) As String
'List the text between the title tags:
Dim regexp As Regex
Dim strMeta As String
Dim strPattern As String
Dim strInPattern As String

'If no description or keywords are found then you may be
'using http-equiv= instead of name= in your meta tags
If InStr(1, LCase(strFile), strMetaStart, 1) = 0 _
And InStr(strMetaStart, "name=") Then
'Swap name= for http-equiv=
strMetaStart = Replace(strMetaStart, "name=", "http-equiv=")
End If

'Build Pattern
strInPattern = "((.|\n)*?)"
strPattern = String.Format("{0}{1}{2}", _
strMetaStart, strInPattern, strMetaEnd)
regexp = New Regex(strPattern, RegexOptions.IgnoreCase)

'Match Pattern
strMeta = regexp.Match(strFile).ToString

'Build Pattern
strInPattern = "(.*?)"
strPattern = String.Format("{0}{1}{2}", _
strMetaStart, strInPattern, strMetaEnd)

'Get Pattern content
strMeta = regexp.Replace(strMeta, strPattern,_
"$1", RegexOptions.IgnoreCase)

Return strMeta
End Function

PagesDataset.vb



This class is used to create and build the DataSet. It consists of two methods and StoreFile. Create method creates the DataSet to store the searched results and the Storefile is responsible for adding records to DataTable in the DataSet.

           '*******************************************************
'
' Create Method - Shared method
'
' Creates a datset for the pages and returns the result
'
'********************************************************
Public Shared Function Create() As DataSet
'Objects are defined
Dim pgDataSet As New DataSet()
Dim keys(1) As DataColumn

'Table is created and added to table collection
pgDataSet.Tables.Add(New DataTable("Pages"))

'Schema of table is defined
pgDataSet.Tables("Pages").Columns.Add("PageId", _
System.Type.GetType("System.Int32"))
pgDataSet.Tables("Pages").Columns.Add("Title",_
System.Type.GetType("System.String"))
pgDataSet.Tables("Pages").Columns.Add("Description", _
System.Type.GetType("System.String"))
pgDataSet.Tables("Pages").Columns.Add("Path", _
System.Type.GetType("System.String"))
pgDataSet.Tables("Pages").Columns.Add("MatchCount", _
System.Type.GetType("System.Int32"))

'PageId is defined as indentity
pgDataSet.Tables("Pages").Columns("PageID").AutoIncrement = True
pgDataSet.Tables("Pages").Columns("PageID").AutoIncrementSeed = 1

'PageId is defined as the primary key
keys(0) = pgDataSet.Tables("Pages").Columns("PageId")
pgDataSet.Tables("Pages").PrimaryKey = keys

Return pgDataSet
End Function

'********************************************************
'
' StoreFile Method - Shared method
'
' Creates a datset for the pages and returns the result
'
'********************************************************
Public Shared Sub StoreFile(ByVal dstPgs As DataSet,_
ByVal srchPg As Searchs.Page)
'Objects are defined
Dim pageRow As DataRow

'New row is created
pageRow = dstPgs.Tables("Pages").NewRow()

'Data is added
pageRow("Title") = srchPg.Title
pageRow("Description") = srchPg.Description
pageRow("Path") = srchPg.Path
pageRow("MatchCount") = srchPg.MatchCount

'Row is added to the dataset
dstPgs.Tables("Pages").Rows.Add(pageRow)
End Sub

CleanHtml.vb



CleanHtml class contains a single public shared method which uses regular expressions to clean the HTML content.

        '*****************************************************
'
' CleanFileContent Method
'
' Subroutine to the clean the file of html content
'
'*****************************************************
Public Shared Function Clean(ByVal Contents As String) As String
Dim regexp As Regex
Dim strPattern As String

strPattern = ""
regexp = New Regex(strPattern, RegexOptions.IgnoreCase)

Contents = regexp.Replace(Contents, _
"<(select|option|script|style|title)(.*?)" & _
">((.|\n)*?)</(SELECT|OPTION|SCRIPT|STYLE|TITLE)>",_
" ", RegexOptions.IgnoreCase)

Contents = regexp.Replace(Contents, "&(nbsp|quot|copy);", "")

'Contents = regexp.Replace(Contents, "<[^>]*>", "")

Contents = regexp.Replace(Contents, "<([\s\S])+?>",_
" ", RegexOptions.IgnoreCase).Replace(" ", " ")

'Contents = regexp.Replace(Contents, "<[^<>]+>",_
" ", RegexOptions.IgnoreCase)

'Contents = regexp.Replace("(<(\w+)[^>]*?>(.*?)</\1>", "$1")

Contents = regexp.Replace(Contents, "\W", " ")

'Trace.Warn("File Contents", Contents)

Return Contents

End Function

Site.vb



Site class consists of shared properties which store the configurations
of the entire site. These properties get their values from the web.config
file using the ConfigurationSettings.AppSettings.


Following are the properties of the site class:





























FilesTypesToSearch Returns the files types you want to search
DynamicFilesTypesToSearch Returns dynamic files to search
BarredFolders Returns the barred folders
EnglishLanguage Returns barred files
BarredFiles Returns a Boolean value whether the language is English
or not.
ApplicationPath Assign and returns the path of the application

        '*************************************************
'
' FilesTypesToSearch ReadOnly Property
'
' Retrieve FilesTypesToSearch of the site
'
'*************************************************
Public Shared ReadOnly Property FilesTypesToSearch() As String
Get
Return ConfigurationSettings.AppSettings("FilesTypesToSearch")
End Get
End Property

'*************************************************

'
' DynamicFilesTypesToSearch ReadOnly Property
'
' Retrieve FilesTypesToSearch of the site
'
'*************************************************
Public Shared ReadOnly Property DynamicFilesTypesToSearch() As String
Get
Return ConfigurationSettings.AppSettings("DynamicFilesTypesToSearch")
End Get
End Property
'*************************************************
'
' BarredFolders ReadOnly Property
'
' Retrieve BarredFolders of the site
'
'*************************************************
Public Shared ReadOnly Property BarredFolders() As String
Get
Return ConfigurationSettings.AppSettings("BarredFolders")
End Get
End Property

'*************************************************
'
' BarredFiles ReadOnly Property
'
' Retrieve BarredFiles of the site
'
'*************************************************
Public Shared ReadOnly Property BarredFiles() As String
Get
Return ConfigurationSettings.AppSettings("BarredFiles")
End Get
End Property

'*************************************************
'
' EnglishLanguage Property
'
' Retrieve EnglishLanguage of the site
'
'*************************************************
Public Shared ReadOnly Property EnglishLanguage() As String
Get
Return ConfigurationSettings.AppSettings("EnglishLanguage")
End Get
End Property

'**********************************************************
'
' ApplicationPath Property
'
'Assign and retrieve ApplicationPath of the site
'
'**********************************************************
Public Property ApplicationPath() As String
Get
Return m_ApplicationPath
End Get
Set(ByVal Value As String)
m_ApplicationPath = Value
End Set
End Property

Web.config


The ASP.NET configuration system features an extensible infrastructure that enables you to define configuration settings at the time your ASP.NET applications are first deployed, so that you can add or revise configuration settings at any time, with minimal impact on operational Web applications and servers. Multiple configuration files, all named Web.config, can appear in multiple directories on an ASP.NET Web application server. Each Web.config file applies configuration settings to its own directory and all child directories below it. As mentioned earlier, the site configurations can be assigned in the web.config file.


<appSettings>

<!-- Place the names of the files types you want searching
in the following line separated by commas -->

<add key="FilesTypesToSearch" value=".htm,.html,.asp,.shtml,.aspx"
/>
<!-- Place the names of the dynamic files types you want
searching in the following line separated by commas -->

<add key="DynamicFilesTypesToSearch" value=".asp,.shtml,.aspx" />

<!-- Place the names of the folders you don't
want searched in the following line separated by commas-->

<add key="BarredFolders"
value="cgi_bin,_bin,bin,_vti_cnf,_notes,images,scripts"
/>

<!-- Place the names of the files you don't want searched in the
following line separated by commas include the file extension-->

<add key="BarredFiles"
value="adminstation.htm,no_allowed.asp,AssemblyInfo.vb,
Global.asax,Global.asax.vb,SiteSearch.aspx"
/>

<!-- Set this boolean to False if you are not using
an English language web site-->

<add key="EnglishLanguage" value="True" />

</appSettings>

How to integrate


The application has been tested with the web form SiteSearch.aspx in the root directory. So my suggestion is that you do the same. Later on, you can try moving it to any subfolder. All my classes I have placed in components folder. You can move them to any folder of your choice.


Points of Interest


When I was working on the project, the question was how to display the results. DataGrid was my choice as we can exploit a lot of its features which are not present in other list controls. Once my question was solved, the next was how to pass content to the DataGrid. DataSet was the only alternative. As I worked further before storing the information in DataSet, I had to move to and fro with bulk information about the page. I decided to use site object to store the information.


One of the authors suggested the following best practices:



  1. The class should be small to have approximately half a dozen methods and properties.
  2. Methods be short with again approximately half a dozen lines.

After carefully analyzing the code and keeping the best practices in mind, I redesigned the code to what it is now.


Histoy


Modified the code to read dynamic pages.






CodeToad Experts

Can't find the answer?
Our Site experts are answering questions for free in the CodeToad forums
Rate this article:     Poor Excellent
View highlighted Comments
User Comments on 'Internal Search Engine'
RELATED ARTICLES
ASP FilesystemObject
by Jeff Anderson
An introduction to the Filesystemobject
ASP GetTempName
by Jeff Anderson
Use the GetTempName method to create a randomly generated temporary file on the server.
ASP Format Date and Time Script
by Jeff Anderson
An ASP script showing the variety of date and time formats possible using the FormatDateTime Function.
ASP OpenTextFile
by Jeff Anderson
An introduction to the OpenTextFile Method of the FileSystemObject
Email validation using Regular Expression
by Jeff Anderson
Using regular expression syntax is an exellent way to thoroughly validate an email. It's possible in ASP.
Add or Subtract Hours in SQL or ASP using DateAdd
by Jeff Anderson
A beginners guide to using the SQL DATEADD function to add or subtract hours. Particularly useful when setting the time displayed on the ASP page to a different time zone (eg when the server is in the US, and the site is for a UK audience).
The asp:radiobutton and asp:radiobuttonlist control
by David Sussman, et al
In HTML, radio buttons are used when we need to make multiple sets of choices available, but we want the user to select only one of them.
The asp:checkbox and asp:checkboxlist control
by David Sussman, et al
Checkboxes are similar to radio buttons, and in HTML, they were used to allow multiple choices from a group of buttons.
ASP FileExists
by Jeff Anderson
An introduction to the FileExistsMethod of the FileSystemObject
Concatenate strings in sql
by Jeff Anderson
A brief introduction to concatenating strings in an sql query (using SQL server or access databases).








Recent Forum Threads
• Global.Mapper.v15.2.3.b060614
• Geometric_Glovius_Pro_v3.6.1
• VERO.SurfCAM.v2014
• Schlumberger.Petrel.V2013.2
• Petrel.V2013.2
• Altair.HyperWorks.v12
• VoluMill.v6.1
• VoluMill.NEXION.6
• VERO.SurfCAM.v2014


Recent Articles
ASP GetTempName
Decode and Encode UTF-8
ASP GetFile
ASP FolderExists
ASP FileExists
ASP OpenTextFile
ASP FilesystemObject
ASP CreateFolder
ASP CreateTextFile
Javascript Get Selected Text


© Copyright codetoad.com 2001-2014